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Executive Summary 



Background of the HPSGP Evaluation 

Passed in 1999, the Public Schools Accountability Act (PSAA) established a results-based 
accountability system in California with specific performance targets for schools. The PSAA 
created a system of rewards and sanctions for meeting or not meeting those targets, and 
established assistance programs for low-performing schools. In 2001, the High Priority Schools 
Grant Program (HPSGP) was established as part of PSAA to provide additional funds to the 
lowest-performing schools in the state, taking the place of the prior Immediate 
Intervention/Underperforming Schools Program (II/USP). 

Priority for participation in the HPSGP was given to the lowest ranked schools in the state, and 
participating schools received $400 per student per year for three years (and a possible fourth 
year depending on progress) to use toward implementing improvement strategies. Schools were 
required to develop an Action Plan (or use one previously developed) to serve as a blueprint for 
the school and community to focus on improving student achievement and meeting growth 
targets. Planning year funds of $50,000 were available to schools to use for the development of 
the Action Plan. Schools not making expected progress at the end of three years would then be 
subject to sanctions. 

In 2005, the California Department of Education (CDE) contracted with the American Institutes 
for Research (AIR) to examine the implementation, impact, costs, and benefits of the HPSGP. 
The CDE identified four primary evaluation questions for the study: ^ 



1) How effectively did participating schools and districts implement the HPSGP? 

2) What has been the overall impact of participation in the HPSGP on school and district 
personnel, parents, and the community, and on school and district organization, policies, 
and practices? 

3) What has been the impact from a school’s participation in HPSGP on student 
performance? 

4) What unintended consequences have resulted from the implementation of the HPSGP? 



* The original contract for this evaluation included another question regarding the growth patterns of schools a) 
funded under both the HPSGP and Immediate Intervention and Underperforming Schools Program (II/USP), and b) 
funded under the HPSGP and the Comprehensive School Reform (CSR) program. This question was subsequently 
dropped by the CDE. 
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This final report summarizes evaluation activities completed during this two-year study and 
presents findings for these research questions drawn from a mixed-methods approach. To 
address the study’s research questions, we: 

■ Analyzed extant data, including student- and school-level achievement data for HPSGP 
and non-HPSGP schools within California, HPSGP Annual Reports and expenditure 
reports for all HPSGP schools, and the California Basic Educational Data System 
(CBEDS), 

■ Conducted case study visits to 16 HPSGP schools in nine districts, 

■ Administered and analyzed data from surveys in 106 HPSGP schools, and 

■ Administered and analyzed data from phone surveys in 49 districts. 



To better isolate the impact of the HPSGP in light of various funding sources, exit criteria, and 
prior participation in other school reform programs, the evaluation limited the analyses to 
schools that received HPSGP implementation funds and did not participate in II/USP or the 
Comprehensive School Reform (CSR) programs — referred to as “HP Only” schools in this 
report. The achievement analysis focused on a subset of the HP Only schools that received 
planning grants and the first round of implementation funding. This subset of schools is referred 
to as “HP Only plus planning.” 



Overview of Findings 

• On average, HPSGP schools showed gains in student performance during the period of 
program implementation. However, the effect of participating in the program on student 
performance was negligible. 



The evaluation explored the relationships between achievement trends and the participation of 
schools in the HPSGP using Academic Performance Index (API) and Standardized Testing and 
Reporting (STAR) Program results. Before interpreting these quantitative results, it is important 
to acknowledge the difficulties inherent in the selection of appropriate comparison schools. 
Because the state purposely selected the lowest-performing schools for the HPSGP, there were 
few schools with comparable levels of academic achievement that did not participate in the 
HPSGP, II/USP, or other school reform programs. As a result, the comparison schools selected 
for these analyses had, on average, slightly higher API scores (at the middle and high school 
level) prior to the program implementation, and appear to serve slightly less challenging 
populations. 

HP Only plus planning schools, on average, showed gains in the percentage of students scoring 
proficient or advanced on the California Standards Test (CST). The percentage increased from 9 
percent to 24 percent in English language arts and from 12 percent to 29 percent in math from 
2001-02 to 2005-06. The averages for comparison schools and all other schools showed similar 
trajectories. 
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When controlling for selected student- and school-level characteristics, the results show that HP 
Only plus planning and comparison schools performed virtually the same over the period of the 
program. This overall finding is supported by analyses of student subgroups and by 
longitudinally linked student data we obtained for this study from a large urban school district in 
the state. 

In summary, while HP Only plus planning schools demonstrated academic progress during the 
period of program implementation, their gains were not statistically different from the gains of 
the comparison schools included in these analyses. 

Using qualitative data collected for this study, we attempted to test the notion that more rigorous 
implementation of HPSGP had resulted in improved outcomes. However, examining the 
relationship between student outcomes and what we observed and heard at the case study sites as 
well as the perceptions of survey respondents did not yield clear results. Though in theory better 
implementation would lead to improved student outcomes, we could not document this 
relationship based on available data. 

The survey data did yield many noteworthy findings, which included: 

■ Although more than 60 percent of school respondents reported that a plan for school 
improvement prominently guided their reform efforts, this reported impression was 
not reflected in measured academic gains. 

■ While nearly half of the school respondents described their external provider support 
as appropriate and effective, nearly 45 percent of the surveyed respondents reported 
that the school did not use, or reported that they did not know if the school used, an 
external provider in the development of the Action Plan, even though it was a 
program requirement. 

■ Although the vast majority of HPSGP school respondents indicated an effective use 
of funds, half expressed concern about the short length of the program, and nearly a 
third reported that the untimely arrival of funds did not permit appropriate planning 
and spending. 

■ Spending on personnel was reported as the most common and the most effective local 
use of HPSGP funds. 

■ HPSGP was perceived as having a major role in student achievement gains, despite 
nearly identical academic performance during this period between HPSGP and non- 
HPSGP comparison schools. 

■ A slight majority of respondents reported a lasting HPSGP impact in areas of school 
capacity. 

■ While 60 percent of school respondents indicated confidence in sustaining the impact 
of HPSGP, only 40 percent said they had been able to find funding to continue these 
reforms. 
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General Observations 

Overall performance of low-performing schools (both those participating and not participating in 
the HPSGP) is improving in an era in which state and federal accountability systems have been 
introduced. The accountability movement, including interventions like the HPSGP, has cast an 
important spotlight on chronically underperforming schools. An expectation is being conveyed to 
state, district, and school administrators that the status quo for these schools is no longer 
acceptable. 

This increased attention paid to the state’s lowest-performing schools is laudable, and has 
yielded some positive results for these schools on average as well as for all schools statewide. 
State and federal accountability efforts have likely made a substantial contribution to this 
improved performance, and it seems likely that in a generic sense, the HPSGP has contributed to 
these overall gains as well. 

At the same time, analyses of school- and student-level achievement for this evaluation show no 
meaningful difference between schools participating in the HPSGP and comparison schools. 
Likewise, two prior evaluations of the II/USP (which was similar in many ways to the HPSGP) 
found that while the program focused attention on student achievement and low-performing 
schools, there appeared to be negligible overall impact on student achievement in participating 
schools. 



Recommendations 

Based on our findings, it appears that a short-term categorical approach to school reform is 
insufficient to overcome much larger system inadequacies that fail to provide the kinds of long- 
term support and assistance needed to substantially and consistently improve student 
performance in the state’s most challenged schools. We suggest terminating categorical 
interventions like the HPSGP in favor of more comprehensive statewide school reform that 
provides long-term administrative and resource support to the state’s lowest-performing schools 
enrolling our most academically challenging students. 

However, we also understand broad-reaching state reform to be an unlikely immediate 
alternative. In the interim, we recommend that the state consider alternative investments to 
bolster the performance of the state’s lowest-performing schools, as opposed to relying on 
II/USP- and HPSGP-type interventions. 

Within this overall context, we offer two categories of recommendations below. We begin with 
general state-level recommendations, irrespective of the future of the HPSGP, and conclude with 
specific improvements to the program that may foster a greater impact. 

General Recommendations 

1. Keep the attention on student learning and low-performing schools 

State and federal standards-based policies have been very successful in capturing the attention of 
the education community and the general public and focusing that attention on student outcomes 
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system- wide and on low-performing schools in particular. We urge that this attention continue. 

2. Consider the resources needed for sustained academic success in iow- 
performing schoois, and ensure that they are present and maintained in these 
schoois and their districts 

The state should identify the resources needed in the state’s most challenging, highest-poverty 
schools, fund them accordingly, and ensure that these resources are allocated effectively by 
districts to schools. 

Since the district was found to be a key intermediary between state-level policy and school-level 
implementation, the state should ensure that districts have the resources to provide the necessary 
assistance and support to their schools, and that they allocate them to low-performing schools as 
needed. 

3. Use data on an ongoing basis to identify the extent to which state-ievei programs 
make an impact, and use these data to inform and aiter state-ievei poiicy and 
programs in support of tow-performing schoois as needed 

As the state sets expectations for schools and districts and encourages them to regularly use data 
as a basis for shaping policy and practice, we suggest the same process for the state. The state 
should attempt to actively determine fairly early on how well state-supported interventions are 
working. 

We recommend early and rigorous assessment of a formative nature that can serve to guide and 
adjust implementation, and that is designed to compile evidence as early as possible about the 
extent to which anticipated outcomes are likely to be forthcoming. External evaluations provide a 
means to gain formative and summative information on programs. However, given the high- 
stakes environment and urgency to improve student outcomes, the state itself should establish 
more mechanisms to review policies regularly, assess what components of its policies are on the 
right track, and adjust policies on an ongoing basis as needed. 

4. Enhance the power of CBEDS 

Several of our case study sites and survey schools exhibited alarming principal turnover, and 
teacher turnover was also noted as a particular challenge to reform efforts. However, we were 
not able to compare this reported turnover to our designated comparison schools or other groups 
of interest, as this critical information is lacking in CBEDS. We recommend enhancing CBEDS 
to include questions on the number of years that principals and teachers have been at their 
current school, and the number of years in that same position in other schools. This enhancement 
to the database would serve as a powerful tool to understand staff turnover and its implications 
for student achievement. 

5. Foster data-driven decision making 

Many of the successful schools we encountered (through our HPSGP case studies, as well as 
evaluations of Proposition 227, II/USP, and high-poverty schools) at least partially attribute this 
success to regular assessments and review of data to drive instruction. Many of these systems 
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were said to be locally developed. The state may want to encourage broader development and 
dissemination of such systems in districts and local schools. Unlike a number of other factors 
that have been repeatedly cited as making a difference in regard to school reform (e.g., strong 
leadership), data-driven decision making may be much easier to replicate. 

6. Recognize the influential role districts play in facilitating or constraining school 
improvement, and incorporate mechanisms into accountability policies to 
encourage positive and productive actions at the district level 

One of the key findings of this study was the potential influence of district context on schools’ 
achievement growth. Although the HPSGP attempted to increase the involvement of the district 
in these reform efforts in relation to the IPUSP, the findings from this study show that there is 
considerable room for improvement. The state’s District Assistance Intervention Team (DAIT) 
process should further clarify the role of districts and counties in regard to assisting the state’s 
lowest-performing schools. 

7 . Consider methods to better align the state and federal accountability systems 

Site-level respondents in this study largely reported that while they consider the API to be a 
better outcome measure, they feel pressure to address AYP targets. This is not surprising given 
that 80 percent of the HP Only schools are in Program Improvement (PI), with nearly half of the 
PI schools in Year 4 or 5 of the sanctions. Given the conflict and confusion associated with two 
overlapping accountability systems, we recommend that the state focus further on their 
alignment. 

8. Develop and foster policies that will strive for strength and continuity of school 
leadership, especially at low-performing schools 

Through the site visits and survey data, we have documented the common problem of excessive 
turnover in the leadership at low-performing schools. Conversely, where schools have appeared 
to thrive under these types of interventions, strong and ongoing school leadership was commonly 
found to be an integral part. While change in leadership may be the catalyst necessary to 
meaningful change, it appears very difficult for meaningful long-term planning and change to 
take hold without subsequent stability of leadership. We believe that a valuable role for the state, 
and a possible alternative investment to the HPSGP, would be to allocate funds for recruiting, 
training, and retaining strong principals in our state’s most challenging schools. 

9. Work with districts to develop learning networks where districts and schools in 
need of improvement can be linked with, and can learn from, districts and schools 
that have been successful in improving outcomes with comparable populations of 
students 

In light of the limited communication reported and evident among schools participating in our 
evaluation, we recommend that the state and districts consider working in tandem to create 
opportunities for districts and schools to learn from one other. This could enhance knowledge 
transfer from schools showing substantial progress under reform efforts over time to schools new 
to and struggling with reform. Such learning networks might feature pairing of schools (“sister” 
schools) or clusters of schools that would collaborate and work together toward the common goal 
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of enhancing student achievement. 

10. Look at other states’ efforts to support their lowest-performing schools. Assess 
what investments they are making toward these ends and the degree to which 
they are experiencing results from these efforts 

As a result of national and state accountability systems across the country, many states are 
experimenting with interventions with the same basic intentions as the HPSGP, i.e., to improve 
performance in their most challenged schools and districts. We suggest an investigation into 
what other states are doing and what evidence they have found in regard to a return on these 
investments. 

11. Require participation in future evaluations 

As a grant precondition for any state program, districts and schools should agree to participate in 
state- approved evaluations of the programs. Soliciting the participation of districts and schools 
for this study took considerable persistence. As the state makes substantial investments in 
programs of this type, a reasonable pre-condition for participation is the state’s right to collect 
data regarding whether this investment is cost effective. 

Specific HPSGP Recommendations 

The following are recommendations if the state opts to continue with HPSGP-type interventions. 

1. Target “failure” early: The CDE should monitor the performance of HPSGP 
schools annually and identify actions for schools that do not meet their API 
growth target in a given year. 

When schools are not showing progress annually (e.g., they do not meet their API growth target 
in a given year), there should be an increase in oversight, such as requiring ramped-up support 
from the district and possibly a required continuing role for the external provider. Conversely, 
when schools are showing progress, it may be advisable to add additional rewards, such as 
relaxed requirements (e.g., increased independence or flexibility to carry over funds beyond the 
final year of the grant). 

2. Enhance the district role: The role of the district should be explicitly enhanced 
and the district should be held accountable for school progress and for 
establishing and maintaining “conditions” for success. 

We recommend that bolstered assurances for which districts will be held accountable be a 
prerequisite for school participation in the HPSGP. The analyses from this study suggest that 
active engagement of districts is an important pre-condition for program success. This 
recommendation mirrors the guidelines developed by the CDE for the second cohort of HPSGP 
schools, which institutes a continuous improvement process facilitated by a District/School 
Liaison Team. The guidance also calls for the Action Plan to demonstrate a clear support role for 
the district in the development and implementation of the plan and shared responsibility for 
school progress. 
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In fostering district accountability, we recommend that the CDE develop a system of rewards 
and sanctions at the district level that are associated with the success or lack thereof of 
participating schools. For example, in regard to the assurances above, district compliance should 
be especially closely monitored in cases where participating schools are not showing success. 
Initially, districts should be reminded of their responsibilities in regard to program 
implementation and that these assurances must be fulfilled to allow continued program 
participation. Ultimately, if districts do not comply and schools are continuing to fail, ongoing 
program funding should be withheld. Rewards for gains in student performance might come in 
the form of increased local discretion. 

3. Improve monitoring: The CDE (perhaps with the assistance of the County Offices 
of Education) should enhance its monitoring of non-achievement-related 
measurements, such as compliance with the district assurances and 
expenditures. 

Along with these district assurances, we recommend regular reporting and monitoring. As the 
CDE is charged with allocating HPSGP funds, they should also be given the responsibility and 
authority to ensure that the program is implemented as designed and to terminate the program in 
a given school or district- wide when this is clearly not the case. 

4. Redesign Annual Report: Collect data necessary to monitor assurances and 
school progress, and review on a regular basis. 

As described above, we recommend enhanced monitoring, and an important step in this direction 
is the modification of the current data collection under this program. The research team did not 
find the current Annual Report data to be particularly helpful in evaluating the program, nor had 
these data been analyzed in any systematic way prior to this evaluation. Changes to the Annual 
Report data collection could make the data more powerful and meaningful for monitoring 
HPSGP schools and districts. 

While our survey collected respondent perceptions about key program components such as the 
external provider and district support, the fact that the evaluation was conducted at the end of 
(and even after) the program made collecting reliable measurements of implementation fidelity a 
challenge. We encourage the state to learn from evaluations of CSR model providers, such as 
High Schools That Work, that use ongoing survey measures to assess the extent to which 
participating schools are implementing the model with fidelity and how that relates to student 
outcomes. We recommend that the CDE redesign the Annual Report as a carefully constructed 
survey instrument that will provide indicators of implementation which can then be used, with 
other measures, to monitor schools as well as assess the relationship between implementation 
and student outcomes. 

5 . Ensure predictable funding: The timing of the funds should be carefully 
considered for the next cohort, with explicit timelines to allow for effective school 
planning and clear expectations regarding a transition phase prior to program 
completion. 

The state and districts should provide clear directives and assurances as to when the funds will 
arrive at the school, how much, and with what degree of flexibility in carry-over. Districts with 
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sufficient resources should support schools in implementing the program (e.g., allow schools to 
plan in the spring/summer) when state funds are delayed, and schools should be allowed time 
extensions in meeting their performance targets if the funds do not arrive at the school on time. 
For instance, if resources do not arrive at the school until mid-year, it may be unreasonable to 
expect that substantial academic growth will be realized through the program in that year. 

To facilitate the continuation of reform, the CDE should provide clear expectations about a 
transition phase. For instance, districts and schools (through the external provider and District- 
School Liaison Team) should submit a transition plan at the beginning of the third year of 
implementation. This plan would assess the reforms/changes attributed to HPSGP funds, identify 
which strategies have been most effective, and identify resources necessary (e.g., financial and 
personnel) to allow the schools to continue key strategies beyond the HPSGP. 

For the second cohort of HPSGP schools, CDE has prohibited annual carry-over. We strongly 
recommend that the CDE reconsider this restriction. While we observed considerable carry-over 
in all years of the program, our case studies suggested that carry-over was an indicator of more 
systemic problems, such as disruption in school leadership. As an alternative, we encourage 
closer monitoring of carry-over, such as requiring schools with substantial carry-over to submit 
an explanation of the reasons and the implications for future planning. 

6. Ensure a supply of qualified external providers statewide, consistently describe 
the nature and duration of their role, and add measurements of their effectiveness 
to the program. 

Study respondents expressed concerns regarding the overall supply of qualified external 
providers. If this component is required as part of the HPSGP, the state has an obligation to be 
more proactive in ensuring an adequate and qualified supply. If the state does not have the 
capacity to develop this pool, then perhaps this component should not be required, or alternative 
options should be allowed. 

In addition, a number of school respondents reported the external provider component as vaguely 
defined. This component also showed substantial variation in implementation. Although the 
external provider role is only required in legislation for the development of the Action Plan, it is 
further described in the second cohort guidelines as to “provide ongoing technical assistance to 
the school site administrative and teaching staff.” This language seems to imply a relationship 
with the external provider for the duration of the grant. The requirement should be fully clarified 
and the supporting language made as consistent as possible. 

Last, the regular cycle of the continuous improvement process described in the second cohort 
guidelines should include an assessment of the effectiveness of the external provider, as currently 
there appears to be no accountability for these individuals who share a large responsibility in 
assisting the lowest-performing schools in the state. 
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Conclusion 

On average, the state’s lowest-performing schools progressed during the period of HPSGP 
implementation. Although the schools participating in this program did not show gains that 
statistically differ from non-participating schools, all of the schools — as well as the state — 
deserve credit for their advances. The findings from this evaluation should not in any way detract 
from these accomplishments. 

The challenge facing the state’s lowest-performing schools are daunting. Many of the educators 
who participated in the site visits and surveys convinced us of their dedication and determination 
in producing a brighter future for their students. It may simply be that the HPSGP was not 
enough. Ongoing systems of supplemental fiscal resources, selective staff placement, and other 
support are needed to substantially impact student outcomes in the state’s most challenged 
schools. 

Given the primary purpose of the program, some may say that the finding of no substantial 
difference in student performance between HPSGP and comparison schools is the only result that 
matters. As this is the third study issued on behalf of the state showing virtually no return in 
terms of enhanced student performance from the HPSGP and its predecessor II/USP, the 
question of whether to continue to invest in HPSGP-type interventions should be carefully 
considered by policy makers. Issues related to the need to improve student performance in the 
state’s most challenged schools will not go away regardless of the future of the HPSGP. 

We recommend that the state’s commitment to low-performing schools not be diminished, but 
enhanced and re-directed. Because the current investments have not fully yielded the desired 
results, the need for a bolstered state commitment to equal educational opportunities for all 
children in California is perhaps greater than ever. 
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Chapter 1 : Introduction 



Background of the Evaluation 

Passed in 1999, the Public Schools Accountability Act (PSAA) established a results-based 
accountability system in California with specific performance targets for schools. The PSAA 
created a system of rewards and sanctions for meeting or not meeting those targets, and 
established assistance programs for low-performing schools. In 2001, the High Priority Schools 
Grant Program (HPSGP) was established as part of PSAA under Assembly Bill (AB) 961, 
Chapter 747, to provide additional funds to the lowest-performing schools in the state, taking the 
place of the prior Immediate Intervention/Underperforming Schools Program (II/USP). In 
addition to the state accountability system, the federal No Child Left Behind Act of 2001 
(NCLB) imposes another set of accountability targets for schools in California. 

While schools in deciles 1-5 were eligible, priority for participation in the HPSGP was given to 
schools ranked in the lowest decile on the state Academic Performance Index (API). A total of 
658 schools statewide participated in the program, receiving over $740 million^ in HPSGP 
implementation funds between 2002-03 and 2005-06. These schools compose 10.3 percent'^ of 
the state’s students and overwhelmingly serve high poverty, high minority, and high English 
learner student populations. 

In 2005, the American Institutes for Research (AIR) was awarded a contract to conduct the 
legislatively mandated evaluation of the HPSGP to examine its implementation, impact, costs, 
and benefits. The California Department of Education (CDE) identified four primary evaluation 
questions for the study 

1) How effectively did participating schools and districts implement the HPSGP? 

2) What has been the overall impact of participation in the HPSGP on school and district 
personnel, parents, and the community, and on school and district organization, policies, 
and practices? 



2 

Each decile represents 10 percent of all schools. The “first” decile refers to the lowest-performing 10 percent of 
schools in terms of API. The “tenth” decile refers to the highest performing 10 percent of schools. The 2001 API 
Base assigned 738 schools, on average, to each decile. 

^ Data obtained from CDE HPSGP Eunding files for 2002-03 through 2005-06. 

Source; California Basic Educational Data System SIE Eiles for 2004-05. 

^ The original contract for this evaluation included another question regarding the growth patterns of schools a) 
funded under both the HPSGP and Immediate Intervention and Underperforming Schools Program (II/USP), and b) 
funded under the HPSGP and the Comprehensive School Reform (CSR) program. This question was subsequently 
dropped by the CDE. 
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3) What has been the impaet from a sehool’s partieipation in the HPSGP on student 
performance based on: 

b. Results of assessments used to determine whether or not schools have made 
significant progress towards meeting their growth targets as specified in the 
PSAA, 

c. Longitudinal analysis of academic performance data of schools participating in 
the HPSGP compared to the academic performance data of all low-performing 
schools in the State, and 

d. Results of disaggregated student performance data for each of the following 
subgroups, as specified in PSAA: 

i. Major racial and ethnic groups 

ii. English language learners 

iii. Students with disabilities 

iv. Students with socioeconomic disadvantages 

4) What unintended consequences have resulted from the implementation of the HPSGP? 



This final report summarizes evaluation activities completed during this two-year study and 
presents findings for the research questions drawn from our case studies, surveys, and analyses 
of achievement and other extant data. 



Methods Used for this Evaluation of the HPSGP 

The HPSGP study design uses a mixed methods approach, including: 

1) Anaiyses of extant data, inciuding student- and schooi-ievei achievement data for HPSGP 
schoois and non-HPSGP schoois within Caiifornia, HPSGP Annuai Reports and 
expenditure reports for aii HPSGP schoois, and the Caiifornia Basic Educationai Data 
System (CBEDS); 

2) In-depth case studies of 16 HPSGP schoois in nine districts; 

3) Web/paper surveys of a sampie of HPSGP schools, and 

4) District administrator phone surveys 

This approach and supporting conceptual framework are explained in more detail in Chapter 
2: Methodology. 



The remaining sections in this chapter provide information about the context for the HPSGP in 
relation to federal and state school improvement and accountability programs, and an overview 
of the specific program components. 
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Overview of Federal and State School Improvement and 
Accountability Programs 

As the HPSGP was not implemented in isolation, we need to consider the program’s 
implementation and outcomes in participating schools in light of prior and concurrent 
participation in other school improvement and accountability programs, including the II/USP, 
Comprehensive School Reform (CSR)^, and NCLB. Nearly half of the schools participating in 
HPSGP participated in one of the three cohorts of II/USP or the CSR program. Given these 
confounding factors in assessing the impact of the program, the analyses in this report include 
schools that participated only in the HPSGP. 

In addition, schools and districts receiving Title I funds must meet Adequate Yearly Progress 
(AYP) targets required by NCLB. Nearly 95 percent of schools that participated only in the 
HPSGP are also in Title 1, and two-thirds are in Program Improvement and subject to corrective 
action and/or restructuring sanctions. In these schools, NCLB may take precedence in the 
school’s focus on improvement. These overlapping accountability systems present confounding 
factors that limit our ability to ascertain the independent effect of the HPSGP. 

No Child Left Behind Act (NCLB) (2001) 

A focus on school accountability has resulted in efforts at both the federal and state levels to 
identify and improve the lowest-performing schools. Since the mid-1990s, states across the 
country have instituted performance-based accountability policies as part of their standards- 
based reforms. In addition, new policies and programs have been put in place to provide 
additional support and resources to low-performing schools. States are now being further 
challenged by the intensified accountability demands of NCLB, which requires states to set AYP 
targets that highlight the gaps in achievement among groups of students and among schools, and 
to develop strategies to support schools that consistently do not meet their AYP targets. 

NCLB uses AYP as an outcome measure to monitor student achievement across schools. In 
California, AYP encompasses several measures and targets, including participation and 
proficiency rates on state assessments including the California Standards Test (CST) and the 
California High School Exit Exam (CAHSEE); the Academic Performance Index (API); and 
graduation rates for high schools. Specific targets are set within each of these categories. All 
schools, districts, and numerically significant subpopulations are expected to meet proficiency 
targets (also known as annual measurable objectives, or AMOs) for English language arts (EEA) 
and mathematics. Eor example, in 2005, for an elementary school to meet AYP, the school 
needed a 95 percent participation rate on statewide assessments, 24.4 percent of students scoring 
proficient on EEA assessments, 26.5 percent proficient on mathematics assessments, and a one 
point growth on the API or a score of at least 590. 

expected percentage of students at or above proficiency has increased, and will continue to 
increase, on a schedule laid out by the state and guided by the federal government. The AYP 



® The federal Comprehensive School Reform (CSR) program (formerly the 1998 Comprehensive School Reform 
Demonstration Program) provides grants for up to three years to support the implementation of comprehensive 
school reform based on research-based effective practices. 
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targets encourage schools to move students from “basic” and “below basic” to “proficient” in 
order to increase the percentage of students in or above this targeted band. To meet AYP, all 
numerically significant subgroups, including English learners and students with disabilities, must 
also meet these expectations. 

Under NCLB, any Title I school is subject to sanctions if it fails to make AYP for two 
consecutive years. At this point, the school enters “Program Improvement” (PI) and remains in 
PI status until it has met AYP for two consecutive years. Local education agencies (LEAs) are 
responsible for providing technical assistance and corrective actions during the PI process. There 
are a broad range of sanction and intervention options that gradually increase in severity each 
year that a school does not make AYP. These interventions include notification to parents that 
the school is in PI status; giving parents the option to change schools; replacement of school 
staff; and ultimately, school restructuring. A series of other instructional modifications and 
support enhancements are required along the way. With NCLB, the LEA is the primary entity 
responsible for taking corrective action with underperforming schools. 

Under NCLB, LEA can also be identified for PI by failing to make LEA AYP for two years in a 
row. An LEA is identified for PI when it does not make AYP in the same content area (ELA or 
Math) and does not meet AYP criteria in the same content area in each grade span, or does not 
make AYP on the same indicator (API or graduation rate), for two consecutive years. Program 
Improvement LEAs are expected to create a plan for improvement and to implement that plan in 
the year after being identified. They face additional corrective actions in the third year. 
Corrective actions could include the replacement of district staff or appointing a state trustee in 
place of the superintendent, among other options. LEAs do not exit PI status until they have 
made AYP for two consecutive years. 

Public Schools Accountability Act (PSA A) (1999) 

Prior to NCLB, C a lifornia instituted its own results-based accountability system. The PSAA 
legislation grew out of recommendations proposed in a 1997 report entitled, “Steering by 
Results.” This report was released by the Awards and Interventions Advisory Committee, a 
committee established by the California Legislature to aid in the development of a plan “for the 
establishment of incentives for the improvement of pupil academic achievement.” In this report, 
the recommendation was made for a “comprehensive program of incentives, positive and 
negative, that would have as its goal an increase in the number of students who meet or exceed 
[the] standards.” 

Governor Gray Davis sponsored and signed the PSAA legislation in 1999 to establish a high- 
stakes accountability system in California that set specific performance targets for schools, a 
system of rewards and sanctions for meeting or not meeting those targets, and assistance 
programs for low-performing schools. The PSAA legislation originally included three major 
components: the API, the II/USP, and the High Performing/Improving Schools Program (also 
known as the Governor’s Performance Award (GPA)). The High Performing/Improving Schools 
Program was an incentive program awarding schools that met their growth targets, showed 
comparable growth among all significant ethnic and economically disadvantaged subgroups, and 



^ LEAs include county offices of education. 
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satisfied testing participation rate requirements; no funds have been appropriated for awards 
since 2002. 

The cornerstone of the state’s accountability system, the API is a numeric index that enables the 
monitoring and comparison of student achievement across schools. The API provides the basis 
for growth targets is used to identify schools for sanctions, interventions, and targeted programs 
like IPUSP and HPSGP. The API is a numeric index assigned to each school, ranging from 200 
to 1000. Initially based solely on the results of the norm-referenced SAT-9 portion of the STAR 
program, calculation of the API now incorporates the CAHSEE and the CST in EEA, 
mathematics, science, and history /social science, and has increased the weight assigned to these 
standards-based measures (the exact weight depends on the grade span of the school and, to a 
lesser degree, on the number of valid scores). 

The State Board of Education set an interim performance API target of 800 for all schools to 
achieve. This goal has dictated the basis for determining yearly individual school API targets. 
Eor a school with an API score below 800, the annual performance target is to grow by five 
percent of the difference between its base API score and 800. Eor a school with an API score of 
800 or above, the target is to maintain a score of at least 800. Eor a school to reach its target, it 
must also show comparable improvement for all numerically significant ethnic and socio- 
economically disadvantaged subgroups.^ As of 2005-06, special education students and English 
learners are included in these subgroup targets. An alternative accountability system has been 
approved to account for schools with fewer than 100 students, special education schools, and 
alternative schools. 



Immediate Intervention/Underperforming Schools Program (ll/USP) (1999- 
2001 ) 

As II/USP was the forerunner upon which many of the HPSGP components are based, it is 
worthwhile to briefly describe this program. The lEUSP was first implemented in the summer of 
1999 when schools scoring in the bottom half of the state’s schools on the SAT-9 for two 
consecutive years (1998 and 1999) were invited to submit an application to participate in the 
program. Cohort 1 included 430 schools, representing a range of grade levels, SAT-9 deciles, 
and geography. Cohort 2 included an additional 430 schools in the fall of 2000, and 430 were 
included as Cohort 3 in the fall of 2001. These Cohort 2 and 3 schools had API scores in the 
lower five deciles and had not met their API growth targets in the previous year. 

Schools that participated in lEUSP made the explicit trade-off of receiving additional resources 
over three years for potential consequences at the end of this period, should those resources not 
result in improved student performance. lEUSP schools received $50,000 for a planning year to 
develop an Action Plan for school improvement with the required assistance of a state- approved 
External Evaluator. They then received funding at a level of $200 per pupil per year to 
implement the Action Plan. 



* This is known as the comparable improvement target and is set at 80 percent of the schoolwide target The target 
for numerically significant subgroups changed for the 2006-07 school year, at which point the subgroup targets were 
the same as the schoolwide target (i.e., 5 percent of the difference between the API base and 800). 
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As participants in the II/USP, these schools were expected to identify barriers to student 
improvement, devise strategies to remove them, and ultimately to show improvement in student 
achievement. Schools that showed no growth in two implementation years became “state- 
monitored” schools and were required to enter into a contract with a School Assistance and 
Intervention Team (SAIT). SAITs are teams of educational consultants — often retired educators, 
and other individuals from private companies, county offices of education, and nonprofit 
organizations — who work with and monitor schools to improve student achievement. The SAIT 
first assesses whether a school has the “essential program components” (EPCs)^ necessary for 
student achievement. In schools where these components are missing, the goal is to implement 
them, focusing on resource allocation and benchmarks for student achievement. 



Overview of the High Priority Schools Grant Program 
(HPSGP) 

In 2001, when NCLB was enacted by the federal government and the II/USP was in its third year 
of implementation, the HPSGP was established as part of PSAA. Although schools ranked in 
deciles 1 - 5 on the API were eligible, priority for participation in the HPSGP was given to 
schools ranked in the lowest decile. Similarly to the II/USP, the HPSGP provided supplemental 
resources for schools to use in the development and implementation of a school improvement 
plan designed to raise student academic achievement, and participating schools were held 
accountable for results. Should HPSGP schools’ improvement efforts not yield sufficient growth 
to meet the state’s API targets, they would face the threat of state sanctions. 

The core components of the HPSGP are: 

1) Targeting of resources to the lowest- performing schools. Schools participating in the 
HPSGP received $400 per student per year for three years (and a possible fourth year 
depending on progress) to use towards implementing improvement strategies. 

2) Action Plan. Schools are required to develop an Action Plan (or modify an existing 
plan) to serve as a blueprint for the school and community to focus on improving 
student achievement and meeting growth targets. 

3) District Monitoring. Districts must monitor the development of the Action Plan and 
report on schools participating in the HPSGP by submitting annual reports that 
account for school characteristics such as instructional materials used, courses 
offered, levels of parental involvement, teacher training, and principal experience. 

4) Sanctions. Schools within the HPSGP are expected to meet API growth targets. 
Failure to make significant growth by 36 months from September 2002 was to result 
in interventions or sanctions. 



9 

The EPCs vary by grade level, but in general include components such as the adoption of state-board adopted 
curricula, AB 75 training for principals, and the implementation of an assessment system to monitor student 
progress. Retrieved June 27, 2006 from http://www.cde.ca.gov/ta/lp/vl/essentialcomp.asp . 

*** Although a second round of HPSGP schools have been approved, this evaluation considers only the first cohort of 
HPSGP (2002-03 to 2004-05, or 2005-06 for schools receiving fourth year funds). 
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The HPSGP and II/USP share many underlying assumptions and program requirements. Both 
seek to improve instruction and student learning by focusing public and educator attention, by 
providing additional resources to help schools improve, and by creating extrinsic incentives 
(sanctions) to motivate change. However, the HPSGP targets funds more narrowly (to the 
lowest-performing schools in decile 1 of the API), doubles the level of funding, provides an 
extended implementation period for schools, more clearly acknowledges the district’s role in 
school improvement by including a district monitoring component, and allows entities other than 
state- approved External Evaluators to provide technical assistance. Several of these changes 
were shaped by lessons learned from the implementation of the lEUSP. Exhibit 1.1 provides a 
side-by-side comparison of the features of the HPSGP and II/USP. 
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Exhibit 1.1. II/USP and HPSGP Comparison 





II/USP 


HPSGP 


Eligibility criteria 


Decile 1 - 5 on API rankings 


Decile 1 - 5 on API rankings 
(Priority for Decile 1) 


Number of cohorts 


3 


1* 


Roie of the district 


■ Approves Action Plan 


■ Participates in the 
development of the Action 
Plan 

■ Approves Action Plan 

■ Submits annual data on 
HPSGP schools 

■ Permitted to serve as an 
external provider 


Impiementation funding 
amounts (per student, per 
year) 


$200 


$400** 


Matching grant 
requirement 


$200 


$200 


Planning year grants 


Yes (required) 


Yes (optional) 


Action Plan 


Required 


Required 


External Provider 


Required. Schools selected from 
state-approved list. 


Required. Schools had more 
flexibility in the selection. 


Implementation period 


2 or 3 years 


3 years 

(4*'^ year funding based on meeting 
targets/making significant growth) 


When progress (for 
“significant growth”) is 
evaluated 


After 2 years 


After 3 years 


Definition of “significant 
growth” 


1 point API growth per year 


10 points combined API growth 
over 3 years, growth must be 
positive 2 out of the 3 years 




After 2 years: 


After 3 years: 


What happened to schools 
after progress is 
evaluated? 


■ Schools that made API 
targets for 2 consecutive 
years exited program; no 
additional funding. 

■ Schools that made significant 
growth 1 out of 2 years 
became under watch, and 
provided 3'^'^ year of funding. 

■ Schools that did not meet 
targets and did not make 
significant growth entered 
SAIT process. 


■ Schools that met targets, or did 
not meet targets but made 
significant growth received a 4'*^ 
year of funding (2005-06) 

■ Schools that did not meet 
targets and did not make 
significant growth entered SAIT 
process. 



* A second cohort has been funded for 2007-08. 

** Variations in funding sources will be described in the following section. 
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One important distinction between the II/USP and HPSGP is the definition of “significant 
growth” -the minimum API growth threshold that schools must achieve in order to avoid 
sanctions. Under the II/USP, schools making significant growth were schools that in a given 
year, made at least a 1 API point gain. Under the HPSGP, “significant growth” was revised to 
encompass a school’s API growth performance over the program’s implementation period. A 
minimum total growth of 10 API points over three years (with growth being positive two out of 
three years) was set as the criteria for receiving an additional year of HPSGP funds. Schools that 
did not make significant growth were subject to state sanctions, which to date has been the SAIT 
process. 



HPSGP Implementation 

The HPSGP was first implemented in 2001 when the state distributed a limited number of 
planning grants to assist schools in the development of an Action Plan. The one-time planning 
grants, valued at $50,000, were optional for schools interested in participating in the HPSGP. To 
ensure that the planning money was targeted to the lowest-performing schools in the state, the 
CDE used the 2000 API Growth score to rank applying schools from lowest to highest, and 
funded the schools in this order until funding was exhausted. Of the 360 planning grants awarded 
in the 2001-02 school year, 78 percent of the schools were ranked in decile 1 of the Base API 
score in 2001, 19 percent were in decile 2, and two schools were in decile 3.*^ 



Exhibit 1.2. 2001 State Rank Distribution of Schools Receiving HPSGP Planning 
Grants, 2001-02 



2001 State Rank 
(From 2001 API 
Base Report)* 


Number of schools 
receiving HPSGP 
planning grants in 
2001 - 02 


Percent of total 
HPSGP planning 
grant schools 


Total number of 
schools in 
California (per 
decile) 


Percent of total 
receiving planning 
grants (per decile) 


Decile 1 


281 


78.1% 


761 


36.9% 


Decile 2 


68 


18.9% 


711 


9.6% 


Decile 3 


2 


0.5% 


730 


0.3% 


Missing Rank 


9 


2.5% 


- 


- 


Total 


360 


100% 







*Note: Although schools were selected for planning grants using the 2000 API Growth score, state decile ranks are 
only Included In API Base reports. Therefore, we present the 2001 state rank. 



In 2002, the first round of HPSGP implementation funds became available. Eligibility for 
receiving these funds was determined by ranking schools from lowest to highest according to the 
2001 API Growth score. The CDE invited 826 of the lowest-ranked schools to submit an 
application. Schools submitting applications that fully met the CDE’s requirements were 
awarded implementation funds according to their rank until state funds were exhausted. See 
Exhibit 1.3 for the specific number and distribution of schools receiving implementation funds. 



" The distinction between Base and Growth can be found at http://www.cde.ca.gov/ta/ac/av/glossarvQ7a.asp . For 
instance, the 2006 Growth API is compared to the 2005 Base API to determine growth in the API from 2005 to 
2006. The 2006 Base API summarizes performance on the spring 2006 STAR Program and California High School 
Exit Examination (CAHSEE). It serves as the baseline score, or starting point, of performance, to be compared to 
the following year’s performance. 
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Due to differences in timing and criteria used for awarding planning grants and implementation 
funds, 36 schools with planning grants did not subsequently receive implementation funds. 

The HPSGP application provided schools with guidelines to follow in the development and 
implementation of the Action Plan. Specifically, it required the participation of the school site- 
council or Action Plan team and the local bargaining representative. In developing the plan, 
schools were also required to use technical assistance from an external provider which could be 
school district personnel, county offices of education, universities, a CDE-approved External 
Evaluator (private provider), or any other person or entity with proven successful expertise 
specific to the challenges in low-performing schools. In addition, schools were required to 
address 14 dimensions of school improvement in the Action Plan, including the identification of 
barriers to academic achievement at the school and district, specification of strategies to address 
these barriers, strategies to focus on literacy with an emphasis on English learners and other 
numerically significant subgroups, and plans for involving teachers in AB 466 training and 
administrators in AB 75 training. Schools were asked to submit a six-page narrative summary of 
their Action Plan as a part of the HPSGP Application. 

Schools selected for the HPSGP received implementation funds amounting to $400 per pupil per 
year (with a $200 matching requirement) over the course of three years. These funds were 
provided to support the activities laid out in the Action Plan. In June 2002, 536 HPSGP schools 
were selected to receive the first round of implementation funds, as shown in Exhibit 1.3. In 
May, June and November of 2003, when a second round of implementation funds became 
available, 98 additional schools were selected for the HPSGP. Although approval of funding 
for these schools was delayed, they were held responsible for making progress during the 2002- 
03 school year. 



Exhibit 1.3. 2001 State Rank Distribution of Schools Receiving HPSGP 
Implementation Funds 



2001 State 
Rank 

(From 2001 
API Base 
Report) 


Number of Schools 
Receiving First 
Round HPSGP 
Impiementation 
Funds <1> 


Number of Schools 
Receiving Second 
Round HPSGP 
Impiementation 
Funds <2> 


Totai Number of 
Schoois in 
Caiifornia (per 
Deciie) 


%of Totai 
Receiving First or 
Second Round 
Impiementation 
Funds (per Decile) 


Decile 1 


504 


47 


761 


72.4% 


Decile 2 


16 


47 


711 


8.9% 


Missing Rank 


16 


4 


- 


- 


Total 


536 


98* 







* This count does not include 17 schools that received the second round of funds from CSR and are regarded as participants in the 
HPSGP. Please refer to Exhibit 1 .4 for further details. 

<1> Selected in June 2002 

<2> Selected in May, June, and November 2003. 



Each participating school received a total funding amount of $400 per pupil based on its total 
enrollment in 2000-01, according to CBEDS. The total funding amount remained the same 
across the three years of funding, irrespective of variations in enrollment. HPSGP 
implementation funds totaling approximately $740 million were awarded between 2002-03 and 



12 



The term “second round” should not be confused with the second cohort of HPSGP schools funded in 2007-08. 
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2005-06 (excluding planning grant funds). The average value of implementation funds across 
four years of funding was approximately $1.1 mi llion per school. 

Both II/USP and Comprehensive School Reform (CSR) schools were allowed to apply to the 
HPSGP. As a result, the implementation of the HPSGP acquired an additional level of 
complexity with varying funding sources and reform programs. Shown in Exhibit 1.4, HPSGP 
schools can be categorized into three groups: those that previously participated in II/USP 
(whether state- or CSR-funded), those that previously participated in CSR, and HPSGP schools 
that did not participate in II/USP or CSR, which we refer to in this report as “HP Only” 
schools.*"^ 



Exhibit 1.4. Number of HPSGP Schools by Program Participation 



Program Participation 


Total Number of 
Schools Receiving 
HPSGP Implementation 
Funds 


As a Percentage of Total 
HPSGP schools 


State-Funded II/USP 


242 


36.8% 


Cohort 1 


48 




Cohort II 


105 




Cohort III 


89 




CSR-Funded II/USP 


48 


7.3% 


Cohort 1 


21 




Cohort II 


20 




Cohort III 


7* 




CSR-Funded HPSGP 


17 * 


2.6% 


HP Only 


351 


53.3% 


Total 


658 


100% 



* These are HPSGP scheols that receive CSR funds In lieu of HPSGP funds, but were still considered to be 
participants in the HPSGP. 



Both state- and CSR-funded II/USP schools were allowed to apply to the HPSGP and could re- 
use a previously created Action Plan in their application. Out of the 658 schools receiving 
HPSGP implementation funds, 44 percent were state- or CSR-funded II/USP schools. II/USP and 
CSR schools that were selected for participation in the HPSGP were funded jointly by both 
programs, receiving $200 from II/USP or CSR, and $200 from HPSGP, for a total of $400 per 
pupil. Regardless of their cohort, jointly funded schools were eligible to receive a maximum of 
three years of implementation funds. In addition to the years in which they had previously 
received II/USP funds. Cohort I jointly funded schools received one year of implementation 
funds under the HPSGP, Cohort 2 schools received two years of implementation funds under the 
HPSGP, and Cohort 3 schools received all three years of implementation funds under the 
HPSGP. 



The exact total value of HPSGP funds for all schools across all years is $740,340,272 and was obtained using data 
from the CDE HPSGP funding files. The average funding value per school was $1,147,142. Including II/USP funds 
for jointly funded schools, the total funding value between 2002-03 and 2005-06 was $1,033,573,792, averaging 
$1,270,960 per school. 

The Year 1 report for this evaluation referred to these schools as “Pure” schools. To make the label more intuitive, 
we call them “HP Only” in this report. 
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A total of 17 schools, referred to as CSR-funded HPSGP schools in Exhibit 1.4, received funding 
from CSR to participate in the HPSGP. These schools received the full amount of $400 per pupil 
from CSR. 

Finally, HP Only schools represent 53 percent of the total number of schools receiving HPSGP 
implementation funds. To better isolate the impact of the HPSGP in light of the various funding 
sources, exit criteria, and prior participation in other school reform programs, the CDE and the 
advisory committee for this evaluation advised that we limit the analyses in this report to HP 
Only schools. The II/USP and CSR are no longer funded, and as future HPSGP funding will no 
longer be split between these three programs, the CDE and committee members believed that 
evaluation efforts should focus only on schools that received all funding through the HPSGP - 
what we refer to as “HP Only” schools in this report. The student-level achievement analysis 
which examines the relationship between program participation and achievement focused on a 
subset of the HP Only schools that received planning grants and first round implementation 
funding (n = 229).^^ This subset of schools is referred to in Chapter 3 as the HP Only plus 
planning schools. 



What are HP Only schools? 

For the purposes of this report, we grouped schools that received HPSGP 
implementation funds into three broad categories: 

• Mixed Program Schools: Schools that were jointly funded by HPSGP and ll/UPS 
or CSR. Among the 658 schools participating in HPSGP, 307 (46.7%) were mixed 
program schoois. 

• HP Only: HPSGP schoois that received implementation funds and did not 
participate in II/USP or CSR. Of the total HPSGP schools, 351 (53.3%) were HP 
Only. 

• HP Only plus planning: These schools are a subset of the HP Only group. These 
HPSGP schools a) did not participate in II/USP or CSR, b) received a planning 
grant, and c) received the first round of implementation funds. All three of these 
criteria must be met. In addition, this group was limited to schools with elementary, 
middle/junior high, and high school type in CBEDS. Of the total HPSGP schools, 
229 (34.8%) were HP Only plus planning. 

Please note that the above schools may be participating in other non-specified 
programs, such as the federal Program Improvement and Reading First. 



As part of this evaluation, the research team consults with an Advisory Committee comprised to state, district, and 
local stakeholders. The purpose of this committee was to provide input on the study design, including data collection 
instruments, sampling plans, and data analysis strategies. 

First round funding is illustrated in Exhibit 1.3. 
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Report Overview 

To address the research questions posed for this evaluation, we conducted case study visits to 16 
schools in 9 districts; administered and analyzed data from surveys in 106 HP Only schools and 
phone surveys in 49 districts; carried out analyses of student achievement and data from CBEDS, 
the HPSGP Annual Reports, and HPSGP expenditure files. The results from these activities are 
presented in the following chapters. In the second chapter, we provide the methodology for the 
study. Chapter 3 explores the relationships between achievement trends and the participation of 
schools in the HPSGP, using API and Standardized Testing and Reporting (STAR) student level 
achievement data for 2001-02 to 2005-06. 

In Chapter 4, we provide important school and district contextual factors primarily drawn from 
our survey results to better understand the conditions in which the HPSGP was implemented. 
Additional analyses of personnel resources presented in this chapter are derived from statewide 
data sources such as the CBEDS, as well as the HPSGP Annual Reports. 

Chapter 5 examines the fundamental implementation components of the HPSGP: the 
development of the Action Plan with the assistance of an External Provider, district participation 
in the development, monitoring and implementation of the Action Plan, and the allocation and 
use of supplemental state funds. 

Chapter 6 reports on the perceived importance and impact of the program on school 
improvement efforts. This chapter closes with an examination of respondents’ views of their 
school’s ability to sustain achievement progress and reforms initiated during the program 
implementation. 

In Chapter 7, we conclude with a discussion of major findings from the study and their 
implications for policy and further research. 
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Chapter 2: Methodology 



This chapter provides an overview of the methodology for the study, including the conceptual 
framework that guided our instrument development and qualitative data analysis. The 
methodology for the student achievement analyses is presented in greater detail in Chapter 3. 



Introduction 

Due to similarities between the HPSGP and its predecessor IPUSP, AIR’s comprehensive 
evaluation of II/USP (O’Day & Bitter, 2003) and the II/USP continuation study (Bitter et ah, 
2005) provide especially useful background information from which to explore the 
implementation and effects of the HPSGP. Our present study draws from this prior work and the 
overall research literature on school improvement to support the development of our conceptual 
framework, data collection instruments, and methodological approach to collect and analyze 
qualitative and quantitative data. 

Building on this prior body of work, the HPSGP study design uses a mixed methods approach 
that includes in-depth case studies, surveys, and analyses of extant data. These methods serve 
two broad goals: The first - assessing the impaet of HPSGP - requires quantifying the 
association between program participation and student achievement. The second - assessing the 
implementation of the program - requires a qualitative understanding of school and district 
organization as well as the attitudes and perceptions of stakeholders towards reform. To address 
the first goal, we used statewide achievement data from various sources to analyze the 
differential growth between a subset of HPSGP and comparison schools from across the state. To 
address the second goal, we conducted case studies of 16 HPSGP schools in 9 districts in Year 1 
of this evaluation (2005-06) and surveyed 106 schools and 49 district administrators in Year 2 
(2006-07). We also used information from the HPSGP Annual Reports and End-of-Year 
Expenditure data (both reporting requirements under HPSGP) as well as personnel data from the 
California Basic Educational Data System (CBEDS) to supplement our understanding of key 
aspects of the program and the context in which the program is implemented. Exhibit 2.2 
summarizes how each of these methods relates to the four evaluation questions. 

During the course of the study, we made additional modifications to our methodology. Several 
adjustments were made pursuant to discussions held during quarterly meetings with the CDE and 
biannual meetings with the HPSGP Advisory Committee. Based on these deliberations, one of 
the major decisions made early in the process was to focus the evaluation on HPSGP schools that 
had not previously participated in the lEUSP or Comprehensive School Reform (CSR). The 
rationale for focusing the evaluation on this subset of HPSGP schools was manifold, but centered 
on the notion that to best understand the implementation and impact of HPSGP as it was 
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intended by policymakers, schools participating simultaneously in multiple reform programs 
(i.e., II/USP or CSR) needed to be excluded. This was particularly reinforced by the fact that 
future HPSGP funding will no longer be split between these three programs. 

For the methodological purposes aforementioned as well as the need to identify sources of 
potentially influential differences between HPSGP schools, we grouped schools that received 
HPSGP implementation funds into three broad categories. Exhibit 2.1 shows how these schools 
relate to each other, as well as to the achievement analyses, case studies, and surveys. 

■ Mixed Program Schools: Schools that were jointly funded by HPSGP and II/UPS or 
CSR. Among the 658 schools participating in HPSGP, 307 (46.7%) were mixed program 
schools. 

■ HP Only: HPSGP schools that received implementation funds and did not participate in 
II/USP or CSR. Of the total HPSGP schools, 351 (53.3%) were HP Only.^’ 

■ HP Only plus planning: These schools are a subset of the HP Only group. These 
HPSGP schools a) did not participate in lEUSP or CSR, b) received a planning grant, and 
c) received the first round of implementation funds. All three of these criteria must be 
met. In addition, this group was limited to schools with elementary, middle/junior high, 
and high school type in CBEDS. Of the total HPSGP schools, 229 (34.8%) were HP Only 
plus planning.'^ 

Exhibit 2.1 : Breakdown of HPSGP Schools 



All HPSGP Schools 
(N = 658, 13%) 




Mixed Program Schools 
(N = 307, 47%) 




All HPSGP Schools 
(N = 658 ) 



The entire population of HP 
Only schools were included in 
our analyses of Expenditure 
Data, HPSGP Annual Reports 
and personnel information frorr 
CBEDS. 



We selected a subset of HP Only schools (shaded below) for the 
achievement and qualitative data analyses. 



Schools in Deciles 1 - 5 
Eligible for Participation in HPSGP 



Achievement Analyses 



HP Only + Planning 
(N = 229, 65% ), . ■ 



Case Study 
Schools 
(N = 16, 5 %)' 



Case Studies and Surveys 

Surveyed Schools 
(N = 106, 30%) 



All HP Only Schools 
(N = 351) 



All HP Only Schools 
(N = 351) 



The Year 1 report for this evaluation referred to these schools as “Pure” schools. To make the label more intuitive, 
we call them “HP Only” in this report. 

** The Year 1 report for this evaluation referred to these schools as “Pure-Pure” schools. To make the label more 
intuitive, we call them “HP Only plus planning” in this report. 
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Exhibit 2.2: Crosswalk between Evaluation Questions and Major Research Tasks 


Methodology 
Evaluation Questions 


Statewide 
Descriptive 
School-Level 
Performance 
Analyses (e.g., 
API targets) 


Statewide 

Student-Level 

Achievement 

Analyses 


Longitudinally 

Linked 

Student-Level 

Achievement 

Analyses 


Case Studies 


School and 
District 
Surveys 


Other Extant 
Data Analyses 


1 . How effectively did participating schools and districts 
implement the HPSGP? 














2. What has been the overall impact of participation in the 
HPSGP on school and district personnel, parents, and the 
community, and on school and district organization, 
policies, and practices? 














3. What has been the impact from a school’s participation in 
HPSGP on student performance based on 


a. Results of assessments used to determine whether or 
not schools have made significant progress towards 
meeting their growth targets as specified in the PSAA 


✓ 












b. Longitudinal analysis of academic performance data of 
schools participating in the HPSGP compared to the 
academic performance data of all low-performing 
schools in the State? 














c. Results of disaggregated student performance data 

for each of the following subgroups, as specified in 
PSAA: 

i. Major racial and ethnic groups 

ii. English language learners 

ill. Students with disabilities 

iv. Students with socioeconomic disadvantages 














4. What unintended consequences have resulted from the 
implementation of the HPSGP 














HPSGP Schools Studied 


HP Only plus 
Planning 


HP Only plus 
Planning 


HP Only plus 
Planning in 
large urban 
district 


HP Only 
(Purposive 
Sample) 


HP Only 
(Random 
Sample) 


HP Only 


Schools studied as a percentage of all HP Only schools 
(n = 351) 

'Note that all analyses excluded 1 0 HP Only schools with non- 
traditional school type categories (i.e. alternative, continuation and 
special education) 


65% 


65% 




5% 


30% 


97% 
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Overview of the Conceptual Framework 

To help anchor the design and development of data collection instruments as well as frame the 
interpretation of findings throughout our study, we developed a conceptual framework (Exhibit 
2.3) that builds on the program’s theory of action and existing research of effective schools. We 
paid particular attention to research that examined contexts that may facilitate achievement 
growth in California’s lower performing schools (EdSource, 2006; Education Trust-West, 2005). 
Research on instructional and organizational capacity and on professional development has 
examined the effects of school characteristics and capacity on student learning (Purkey & Smith, 
1983; Eevine & Eezotte, 1990; O’Day, Goertz, & Eloden, 1995; Newmann & Wehlage, 1995; 
Mohrman & Eawler, 1996; Williams, Kirst, & Haertel, 2005). As mentioned, prior AIR 
evaluations of the II/USP (O’Day & Bitter, 2003, and Bitter et ah, 2005) also contributed to our 
framework. 



Exhibit 2.3. Conceptual Framework for the HPSGP Evaluation 




In regards to HPSGP, the conceptual framework focuses on its primary components: the content 
and effectiveness of the school Action Plan, the external provider; the targeting of resources to 
the lowest performing school, and district support and contribution to the implementation of 
strategies for improvement. These components were designed to guide, finance, and motivate the 
school’s implementation of a coherent set of strategies for school improvement. The intermediate 
outcomes targeted by HPSGP included school capacity (including teacher and principal training 
and qualifications) and instruction (specifically the use of SBE-adopted and standards-based 
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instructional materials). While we were not able to explore each of these factors in depth, the 
conceptual framework helped guide our qualitative analysis efforts. Using this framework, we 
focused our efforts on delineating the strategies that underlie factors for “growth” among HPSGP 
schools, in order to inform policy recommendations on how to foster these factors through state 
and local policy. 

In terms of contextual factors affecting schools, this study focused on the district’s mediating 
role in the implementation of the program and school improvement strategies. A strong district 
effect was found in prior evaluations of the IPUSP (O’Day & Bitter, 2003; Bitter et ah, 2005), 
which was reinforced by the variability in the district role in the case studies completed in the 
first year of this evaluation. As mentioned previously, overlapping school accountability policies, 
primarily II/USP and CSR, had significant implications for our research design. Another salient 
and influential contextual factor affecting HPSGP schools in recent years, however, is the federal 
NCLB Act. Since 2002, both schools and districts are expected to meet Annual Yearly Progress 
(AYP) targets. Many HPSGP schools were identified for Program Improvement during the 
implementation of HPSGP as a result of failure to meet their AYP targets, and some were subject 
to corrective action and/or restructuring sanctions as a part of this process. 



Overview of Methodology 

This section provides a more detailed overview of the methods used in this study. The first part 
of this section outlines the achievement analyses used to estimate the impact of the HPSGP while 
the second part describes the methods used to qualitatively assess the program’s implementation 
and impact. As noted previously, the scope of the evaluation was limited to HPSGP schools that 
did not participate in II/USP or CSR (“HP Only”), and the achievement analysis was further 
confined to HP Only schools that received planning grant and the first round of implementation 
funds (“HP Only plus planning”). 

Achievement Analysis 

To accurately and reliably estimate the impact of the program for HP Only plus planning 
schools, we needed to compare the performance of students attending those schools against the 
performance of students in a group of similar sehools that did not participate in the program. The 
selection of comparison schools was the first, most critical, and perhaps most challenging task of 
the achievement analysis component of this study and is explained in more detail in Chapter 3. 
With the exception of the longitudinally linked student level analysis, the comparison group was 
the same for each of the following sets of analyses. 

Statewide School-Level Performance Trends 

We conducted an initial assessment of performance between HP Only plus planning and 
comparison schools using outcome measures developed for the state and federal accountability 
systems. Although only descriptive in nature, these school-level analyses provided us with 
important context that enabled us to gauge the performance levels of HP Only plus planning 
schools relative to comparison schools and the state as a whole. Examining the state system, we 
looked at the percentage of schools meeting the schoolwide API growth targets and the 
distribution of schools by state rank from 1999-00 through 2005-06. To assess progress under the 
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federal system, we looked at the percentage of schools meeting all of the AYP targets from 
2002-03 through 2005-06 as well as the percentage of students scoring proficient or above in the 
California Standards Test (CST), the metric upon which NCLB’s Annual Measurable Objectives 
are set. 

Statewide Student-Level Analyses 

In order to arrive at a more precise estimate of the relationship between participation in the 
HPSGP and student outcomes, we conducted a second set of analyses using student-level 
assessment data collected under the Standardized Testing and Reporting (STAR) system. Using 
STAR assessments enabled us to control for key student demographic variables known to 
contribute to achievement (such as parent education). Controlling for individual background 
characteristics was important because students were not assigned at random to HP Only plus 
planning and comparison schools, and failure to take these characteristics into account could lead 
to biased estimates of the impact of the program. 

These analyses used statewide Stanford Achievement Test (SAT-9) results in reading, 
mathematics, and language from 1998 through 2002, California Achievement Test (CAT/6) 
results in reading, mathematics, and language from 2003 to 2004, and California Standards Test 
(CST) results in English language arts and mathematics from 2002 to 2006. One of the 
challenges encountered in conducting this analysis resulted from having to take into account 
changes in the assessments used to measure student outcomes under STAR: in 2002-03, the 
SAT-9 was replaced by the CAT/6 and an additional test, the CST, was introduced in 2002. The 
approach taken to address these changes is explained in more detail in Chapter 3. 

Using a Hierarchical Linear Modeling (HEM) framework, we modeled student performance in 
each subject area as a function of time, student characteristics (such as socioeconomic status) and 
school characteristics (such as participation in HPSGP). Eor the purpose of this evaluation, our 
main interest was the marginal contribution of the HPSGP to student outcomes. 

Statewide Student-Level Analyses by Student Subgroups 

As requested by the evaluation questions, additional analyses of student-level outcomes were 
conducted using disaggregated data for several sub-populations: English learners (EEs), students 
eligible for free or reduced price lunch, Hispanic, African-American, Asian, and special 
education students. Similar to the statewide student- level analyses described previously, we 
assessed differences in performance between HP Only plus planning and comparison schools for 
a given student subgroup. 

Longitudinally Linked Student Level Analysis of Data from a Large Urban School 
District 

Lastly, we complemented these statewide analyses by estimating the impact of HPSGP using a 
longitudinally linked student level dataset provided by a large urban school district in the state. 
As we were able to follow students over time, we could study the individual performance 
trajectories of students in HP Only plus planning and comparison schools to estimate whether 
differences between those trajectories were a result of their school’s participation in HPSGP. 
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Other Data Analysis 

To further inform the research questions, we drew upon school staffing data from the California 
Basic Educational Data System (CBEDS) for all schools in the state, as well as the HPSGP 
Annual Report data and Expenditure Eiles. Using CBEDS Professional Assignment Information 
Eorm files, we examined the personnel resources in HP Only schools in 2001-02 (i.e., the 
planning grant year) and 2005-06 (i.e., the fourth year of implementation), in relation to 
comparison schools and all other schools statewide. Districts with HPSGP schools are required 
by the program to submit annual data on various elements relating to the program. We used these 
Annual Report data on principal experience in HP Only schools to provide additional context on 
principal turnover in participating schools. While the research team reviewed other data elements 
in these reports, they were not found particularly helpful for evaluation purposes. The HPSGP 
Expenditure Eiles were used to examine the distribution of spending across several expenditure 
categories as well as the amount of unused program funds that were carried over on a year to 
year basis. 

Qualitative Data Collection and Analysis 

To inform evaluation questions 1, 2 and 4 regarding the overall impact, implementation and 
unintended consequences of HPSGP, we needed to gather first-hand data from key stakeholders. 
Our approach consisted of visits to 16 HP Only schools and their districts during the 2005-06 
school year, followed by a large scale survey effort during the 2006-07 school year. The 
following sections describe each data collection in detail, including the sampling, recruiting 
participants, and collecting and analyzing data. 

Year 1 Case Studies 

By visiting schools in person, our primary goal was to better understand the context in which 
HPSGP schools and districts operate. We also wanted to explore the implementation and impact 
of the program in schools that experienced some degree of improvement compared to those 
showing little or no improvement over the course of the HPSGP implementation period. 
Interviews with key personnel at the school site, external assistance providers and district 
personnel associated with the school, as well as focus groups, were designed to inform the 
following: 

1 . Stakeholders’ reform-related attitudes (e.g., commitment to the reform, trust in the processes 
and leadership guiding the reform) and their motivation to initiate, participate, and sustain 
changes that may be necessary to improve student achievement; 

2. Strategies used to create positive teaching and learning environments; 

3. Types and intensity of professional development opportunities and other supports to teachers, 
administrators, and staff; 

4. Eactors that facilitated or hindered the implementation of the HPSGP and school reform 
efforts; and 
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5. Role of the district and external provider in providing technical support and assistance in 
schoolwide reform and improvement, as well as district strategies and supports for HPSGP 
schools. 

Selection of the Case Study Sample 

As noted above, we sought to purposely select improving and non-improving schools as a key 
feature of our sample design. Our primary selection criterion was centered on whether or not 
schools had met their annual growth targets over the first three implementation years (2002-03 
through 2004-05) and a relatively high or low cumulative API growth over the last two of those 
years (2003-04 and 2004-05). Although some schools received planning grants for 2001-02, the 
first implementation funding was not released until June 2002. Accordingly, we treated 2002-03 
as the first year of participation in the HPSGP and identified schools meeting or not meeting the 
API growth targets in that year and onwards. Our definitions for these schools are as follows: 

• Consistent Growth Schools are defined as those meeting API schoolwide and comparable 
improvement growth targets every year, starting with the first year of the implementation 
of the HPSGP. In order to target higher performers within this group of schools, we gave 
preference to schools that demonstrated relative high cumulative API growth across 2004 
and 2005. 

• Recent Low Growth or No Growth Schools are defined as those that did not meet both 
API growth targets in 2004 and 2005. Within this group, we sampled schools that 
demonstrated negative or relative low cumulative growth across 2004 and 2005. 

In addition, our sample was selected to ensure that it represented key features of the broader 
population of HP Only schools. For practical reasons, we attempted to cluster schools within 
districts. In total, we selected 16 schools: 6 elementary schools, 4 middle schools and 6 high 
schools. Of these, half were consistent growth schools and the other half were recent low growth 
or no growth schools. 

Case study data collection 

During the 1 to 2 day school visit, we collected in-depth qualitative data through interviews and 
focus groups with a broad range of stakeholders associated with the schools, including district- 
and school-level staff, school board members, school site council members, external assistance 
providers, parents, and students. Site visits were designed to be conducted with one day spent at 
each school site, followed by a half day spent at the affiliated district office. At the district, we 
conducted interviews with up to two administrators (including individuals responsible for the 
oversight of state and federal intervention programs and/or those with expertise in curriculum 
and instruction, assessment and evaluation), up to two school board members, and external 
assistance providers affiliated with each of the case study schools. 

At each school site visited in Year 1, on average, we interviewed the principal, four teachers, and 
two paraprofessionals. In addition, we conducted separate focus groups with teachers, parents 
(parent focus groups were conducted in either English or Spanish, with additional translation 
services provided either by research or school staff as needed). School Site Council members, 
and students (at the middle and high school level only). Two- to three-person research teams 
conducted the data collection activities, and we requested permission to audio-tape to ensure 
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accuracy of notes (these tapes were not transcribed). The researeh team also conducted 
approximately 30-minute observations of interviewed teachers in their classrooms. Though the 
limited number and duration of classroom observations could not allow us to formally study 
variations in instructional practices, they provided valuable eontextual data for eaeh sehool. All 
respondents involved in interview, foeus groups, and classroom observations were given 
assurances that neither individual nor school names would be identified in the report. 
Additionally, the HPSGP applications submitted by schools were collected and reviewed for 
eaeh ease study sehool prior to the site visits to provide contextual information prior to our visits 
to the schools. 

The case study data collection instruments were designed to address the evaluation questions 
while not burdening respondents. As a first step, we reviewed relevant literature and refined our 
eoneeptual framework to identify key constructs and variables at the distriet, sehool, and 
elassroom levels. Based on literature of best praetiees of high-performing sehools (just4kids.org; 
Kannapel et ah, 2005), findings from the II/USP continuation study and instruments previously 
developed for other studies (for example, the II/USP and Proposition 227), the HPSGP ease 
study instruments were organized around 6 broad themes: leadership and capaeity building 
(including professional development); sehool eulture; perspeetive on HPSGP implementation, 
sustainability, and consequences; school strategies implemented to improve student achievement; 
district role and support for sehool reform efforts; and observations regarding student and school 
outcomes. 

To the fullest extent possible, we attempted to triangulate data so that when possible, information 
gathered was not solely based on a single souree. The resulting eategories of instruments are as 
follows: 

■ Interviews with District Administrators, School Board Members, External Assistance 
Providers, Principals, and Teaching Staff: A major focus of these interviews was to 
understand what factors facilitated and/or hindered the implementation of the HPSGP and 
school reform efforts, as well as how this occurred and why. 

■ School Site Council, Teacher, Parent Focus Groups: School site council member, teacher, and 
parent focus group protocols were designed to reveal their involvement in school practices 
and policies, their understanding of the resources and assistance provided to their school 
through the HPSGP, and any observed effects of the HPSGP. 

■ Student Focus Groups: Student focus groups included questions on the school climate and 
culture, what expectations they and their teachers have of their performance and their future, 
the degree to which schools support parent involvement, the challenges that students face and 
how the school addresses those challenges. 

■ Classroom Observations: Classroom observation protocols were designed to provide a 
snapshot of activities in the classroom useful for providing context to the above data 
collection activities. They focus on classroom environment, lesson content, assessment 
activities, instructional resources, and strategies. 
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In addition, at the end of each interview and focus group (excluding student focus groups), we 
asked each respondent to rate the effectiveness of their school’s improvement strategies, their 
confidence in sustaining improvement beyond the program’s end, the helpfulness of the planning 
year and the HPSGP overall, as well as the support of the district. 

Year 2 School and District Surveys 

While the case studies can be characterized as exploratory in nature, the surveys were designed 
to key in on the important themes that emerged from the case study findings. 

Selection of the Survey Sample 

The survey sample, in contrast to the purposive case study sample, was designed so that results 
could be generalized to the broader population of HP Only schools. In selecting this sample, we 
had two objectives - 1) to stratify by school type to ensure a large enough number of elementary, 
middle, and high schools to conduct meaningful statistical analyses by school type, and 2) over- 
sample to account for inevitable district and school declines to participate in the survey effort. As 
a result, we first set a desired target (based on statistical considerations) of obtaining 90 
completed school surveys (30 from each school type - elementary, middle and high) and 
conducting 45 district surveys. With this target in mind, we over-sampled for a total of 162 
schools from 65 different districts. This random sample represented 46 percent of the 351 HP 
Only schools. 

We randomly selected 162 schools from the larger population of 325 HP Only schools. We did 
not include schools that had previously participated in our case studies so as to not overburden 
them with a second round of data collection requests. In addition, alternative, special education 
and continuation schools were not considered for the sample. 

We decided jointly with the CDE and the HPSGP Advisory Panel to distribute one survey per 
school. Although the survey was targeted to principals, we recognized that with high levels of 
staff turnover, we would need to be flexible in our approach. As a result, our survey instructions 
noted that principals could seek the support of staff with more knowledge or experience related 
to the HPSGP. To make completing the survey as convenient as possible, we offered respondents 
two alternatives: complete a hard copy of the survey and return it to AIR with a pre-paid 
envelope or complete and submit an electronic version of the survey online. 

At the district level, we identified administrators knowledgeable about the implementation of 
HPSGP and conducted an hour-long survey with them over the phone. To encourage both district 
and school participation, we designed a multi-step strategy which is described in more detail 
below. 

District survey recruitment 

First, using contact information provided by the CDE, we sent a package on November 30, 2006 
via FedEx to the superintendents of all 65 sampled districts. The package contained a letter from 
State Superintendent Jack O’Connell endorsing the evaluation and a letter from AIR. The 
documents described the goals and main features of the study and encouraged their participation. 
The AIR data collection team proceeded by following up with district superintendents or their 



The total count of 325 excludes the 16 case studies and 10 alternative, special education, and continuation schools. 
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designees through phone calls, fax messages, and emails. Nine districts required a formal 
application for conducting research with schools. For these districts (which tended to be larger, 
urban districts), we prepared and submitted applications that followed their guidelines. 

After a number of weeks of seeking approvals, 7 districts refused to participate and 4 districts 
were not responsive to our contact efforts. With support from the CDE, we were able to obtain 
the approval from two of the districts that had originally declined. Among the districts that did 
not grant us permission to contact their schools, the reasons cited included new school 
administration, major changes at the school or district (such as restructuring under Program 
Improvement) and reluctance to overburden school or district staff. In total, 21 of the sampled 
schools (5 elementary, 8 middle and 6 high schools) were dropped from the study due to district- 
level refusals or non-response. 

Of the 99 districts with HP Only schools, 65 were sampled and of these a total of 56 gave AIR 
permission to administer the school surveys. 

School survey recruitment 

The 56 districts that agreed to support AIR’s survey effort represented 141 of the sampled 
schools. Between January 29 and February 22, 2007, we mailed these schools a letter from State 
Superintendent Jack O’Connell, a description of the study, instructions to complete the survey, 
and a copy of the survey instrument. Following this mailing, the AIR data collection team made 
a strategic and concerted effort to contact (by phone, fax, and e-mail) the principals of the 
sampled schools to confirm the receipt of the survey, encourage their participation, and address 
any of their concerns or questions. 

Perhaps the biggest challenge of the school level data collection was simply getting in direct 
contact with the principal. Another challenge was turnover; some of the principals were new to 
the school and were thus unfamiliar with the program and its implementation. As a result, our 
strategy shifted by attempting to communicate not only with the principal but with other 
administrators (such as the assistant principal), coordinators (such as the Title I coordinator), 
and, in some cases, teachers. We also attempted, to the extent possible, to leverage already 
developed district-school relationships. In some cases, our district contact offered to serve as a 
liaison between the AIR data collection team and the schools. In other cases, where possible, we 
recommended that our district contact the selected schools to remind them about the survey and 
encourage its completion. District support was an invaluable resource during our data collection 
effort. 

Concerned with the limited number of high school responses, towards the end of our data 
collection period, we asked the CDE to provide additional support by contacting the remaining 
sampled high schools that had not returned the survey. With their support, we were able to secure 
participation from two other high schools. In total, 37 of the sample schools (20 elementary, 26 
middle and 12 high schools) did not return the survey. 

School survey response rates 

Exhibit 2.4 displays the overall school response rates to the survey along with the response rates 
by school type and urbanicity. The exhibit displays two response rates, response rate A is based 
on column A (the total number of HP Only schools sampled) and response rate B is based on 
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column B (the total number of surveys sent to schools). While response rate A reflects school 
attrition due to school non-response and district refusals, response rate B reflects school attrition 
due only to school non-response. Of the 141 schools that received a survey, we obtained an 
overall response rate of 75 percent. The response rate by school type is somewhat varied. In 
absolute terms, we received the least number of completed surveys from high schools (n = 28) 
compared with elementary (n = 44) and middle schools (n = 34). However, high schools had the 
highest response rate relative to the other two school types (85 vs. 76 and 68 percent for 
elementary and middle schools, respectively), due to the relatively smaller total number of HP 
Only high schools. 



Exhibit 2.4. Schooi Survey Sample, Response Rate by School Type and Urbanicity 



Total 
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HP Only 
Schools in 
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HP Only 


HP Only 
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Schools 


Schools 


Rate A 


Rate B 


Schools 
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<1> 


Approval 


Responded 











A 


B 


C 


C/A 


C/B 


Overall 


325 


162 


141 


106 


65% 


75% 


By School Type 


Elementary 


223 


63 


58 


44 


70% 


76% 


Middle 


62 


60 


50 


34 


57% 


68% 


High 


40 


39 


33 


28 


72% 


85% 


By Urbanicity 


Urban 


199 


88 


82 


63 


72% 


77% 


Suburban 


134 


67 


52 


36 


54% 


69% 


Small Town or Rural 


16 


6 


6 


6 


100% 


100% 



<1> These counts do not include 16 case study schools and 10 schools that had an alternative, special education, 
or continuation school type. 



70 

As shown in Exhibit 2.5, the majority of the respondents to the school survey were principals. 
Other respondents included assistant principals or other administrators (9.4 percent), 
coordinators or teachers on special assignment (7.5 percent), and former principals that currently 
serve as district administrators (1.9 percent). 



Exhibit 2.5. School Survey Respondents by Type 



Staff Position 


Count 

(N) 


Percent 

(%) 


Principal 


79 


74.5% 


Assistant Principal/ Administrator 


10 


9.4% 


Coordinator or Teacher on special assignment (e.g. Title 1 Coordinator) 


8 


7.5% 


District Administrator (Ex - Prinoipal) 


2 


1 .9% 


Other* 


5 


4.7% 


Missing 


2 


1 .9% 



‘Other Includes: Executive Director/ Project Director/ Principal Designee/ Language Arts Coach 



Principal respondents reported, on average, 6.7 years at the surveyed school (in any capacity), and 4.5 years as 
being the principal of the surveyed school. On average, the principal respondents have been a principal for 11.1 
years in total (at current and any other school). 
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While we obtained a high response rate (75 percent), we were concerned about selection bias 
stemming from possible systematic differences between schools that agreed to participate and 
those that declined (or districts declined on their behalf) that would affect the representativeness 
of the returned data. For example, it is possible that participating sites may be better organized, 
more successful, and positive about the program that non-study participants. 

While we obtained a high response rate (75 percent), we were concerned about selection bias 
stemming from possible systematic differences between schools that agreed to participate and 
those that declined (or districts declined on their behalf) that would affect the representativeness 
of the returned data. For example, it is possible that participating sites may be better organized, 
more successful, and positive about the program that non-study participants. 

To assess possible bias, we conducted two analyses. First, we compared schools returning the 
survey with the overall population from which they were sampled, across a variety of key 
characteristics. This allows us to test whether the returned survey data appear to be 
representative of the overall population of HP Only schools. To supplement this analysis, we also 
examined schools in our sample that submitted and did not submit the surveys. Please refer to 
Technical Appendix B for results of both analyses. 

When comparing the participating schools to the overall population (analysis 1), we found no 
statistically significant differences within each school level in student demographic 
characteristics, such as percentage of English learners and students eligible for free and reduced 
price lunch. With respect to academic performance, elementary schools participating in the 
survey had a statistically significantly higher average 2006 API Base (653 points) in relation to 
that for the overall population of HP Only elementary schools (640). On all other measured 
performance dimensions, including the percentage of schools in Program Improvement, there 
were no statistically significant differences.^^ 

Among the original sample (analysis 2), there were no statistically significant differences 
between schools that submitted the survey and those that did not, in terms of student 
demographics and academic performance (2006 API Growth, 2005 API Base, 2005-06 API 
Growth, state rank, and similar school state rank). A statistically significantly greater percentage 
of participating middle schools were in Program Improvement (97 percent), in relation to non- 
participating middle schools (83 percent). However, this difference did not appear to affect 
comparability between participating schools and the overall population of HP Only schools (as 
there were no statistical differences in the percentage of PI schools between participating schools 
and the overall population). 

To supplement analysis 2, we obtained the number of years of principal experience at the school 
site for 31 of the 35 schools that received the survey but did not respond, as well as from the 98 
schools that answered the survey question on principal experience. Principals in schools that 



At the 5 percent confidence level. 

These dimensions included the 2006 API Growth, 2005 API Base, 2005-06 API Growth, state rank, and similar 
school state rank 

At the 10 percent confidence level. 

CBEDS does not collect this information. The latest HPSGP Annual Report (2005-06) was missing over a third of 
the HP Only schools and therefore was not analyzed for this purpose. 
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returned the survey had been at the site, on average, more than a year longer than principals at 
the non-responsive sites. In addition, principals at the responding schools had a median of four 
years at the school sites as compared to two years for the non-respondents. Greater longevity of 
leadership at participating schools may have influenced some of the survey responses. As noted 
previously, 21 schools were dropped from the sample due to district-level refusals or non- 
response; stated reasons for declining included a new school administration and restructuring. 
However, without being able to analyze principal longevity for the population of all HP Only 
schools, we do not know if this would introduce bias. 

In summary, there may be potential sources of bias in terms of a) participating elementary 
schools having higher performance than the overall population, b) longer principal tenure at 
schools returning the survey in relation to the non-responsive sites, and c) district declines on 
behalf of the schools to participate due to new staff or school restructuring. However, in each 
case, the likely direction of bias is unclear. 

Exhibit 2.6 displays the total number of districts represented by the school survey sample, the 
number that agreed to allow schools to participate in the study and the number that participated 
in the district phone survey. It is important to note that we pursued district phone surveys even if 
we did not ultimately obtain surveys from the sampled schools in the district. Of the 65 districts 
that were sampled, we surveyed administrators from 45 of them (69 percent.) Realizing the need 
for additional context in one large urban district, we decided to pursue additional interviews with 
administrators in its local districts. Including these local district surveys, in total we conducted 
49 district- level surveys. Across district types response rates were similar; however, with regards 
to total district enrollment we obtained a relatively higher response rate from larger districts - 
those with enrollments between 10,000 and 30,000 and those with more than 30,000 students had 
response rates of 70 and 83 percent, respectively. Moderately sized districts - districts whose 
enrollment range from 1,000 to 10,000 and small districts with less than 1,000 students, had 
response rates of 61 and 50 percent, respectively. 



Exhibit 2.6. District Survey Sample, Response Rate by District Type and District 
Enrollment 





N of Districts with 
HP Only Schools 
(Pool of Eligible 
Districts) 


N of Districts 
in Sample 


N of Districts 
approving 
participation 


N of Districts 
that 

participated In 
phone survey 


Response Rate 






A 


B 


C 


C/A 


Overall 


99 


65 


56 


45* 


69% 


By District Type: 


Elementary 


31 


18 


16 


11 


61% 


High 


9 


6 


5 


4 


67% 


Unified 


59 


41 


35 


30 


73% 


By District Enrollment: 


greater than 30,000 


18 


18 


18 


15 


83% 


between 1 0,000 and 30,000 


30 


23 


18 


16 


70% 


between 1,000 and 10,000 


41 


18 


15 


11 


61% 


less than 1 ,000 


10 


6 


5 


3 


50% 



*Thls number does not Include 4 local districts interviewed in a large district. 



An average 4.5 years at participating sites in comparison to 3.2 years at the non-participating sites. This difference 
was statistically significant at the 10 percent confidence level. 
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Relative to school survey respondents, district respondents were more diverse in terms of their 
roles and responsibilities. As Exhibit 2.7 shows, 14 percent of the district respondents were 
superintendents; 22 percent were either assistant, associate, or deputy superintendents; 37 
percent were department directors, and 27 were coordinators. 



Exhibit 2.7. District Survey Respondents by Type 



Staff Position 


Count 

(N) 


Percent 

(%) 


Superintendent 


7 


14% 


Assistant, Associate or Deputy Superintendent 


11 


22% 


Department Director* 


18 


37% 


Coordinator/ Administrator** 


13 


27% 



‘Examples Include: Directors of Curriculum and Instruction, State and Federal Programs 
“Examples Include: Coordinators of Accountability, Grants, Compliance, Categorical Programs 



School and District Survey Protocols 

The development of the school survey protocol was informed primarily by the findings of the 
Year 1 case studies and was designed to measure perceptions of district support, school capacity 
and the implementation and impact of HPSGP and its primary components. The survey was 
organized around several themes: the overall context of the district, school capacity, staffing and 
administration, level of understanding of the HPSGP, HPSGP funding and use of resources, 
HPSGP external provider, district support under the HPSGP, observations of change, 
sustainability of reform and program assessment. Individual questions under each theme were 
refined pursuant to feedback from the CDE and a series of pilots that were conducted with 
principals from HP Only schools not in our sample. All of the survey questions were closed- 
ended with the exception of five open-ended questions in the survey’s narrative section. The 
purpose of these open-ended questions was to create an opportunity to obtain responses that 
would help contextualize respondents’ answers as well as capture themes that may not have been 
specifically asked about in the closed ended questions. The final school survey protocol is in 
Technical Appendix E. 

The district survey, while similar in length and scope to the school survey, was administered 
differently, and included a set of topics aligned with the roles and responsibilities of the district. 
The survey was organized around several categories: administrator background, district context, 
impact of the HPSGP, HPSGP external provider, district challenges, sustainability and 
conclusions. Some of the topics exclusively explored in the district survey, were the roles played 
by the school board and county office of education, policies to allocate staff and quality of staff 
in HP Only schools. Where possible, we tried to use the same items as in the school survey, in 
order to triangulate responses between school and district respondents. Given that we spoke 
directly with district respondents, the survey built in opportunities for interviewers to ask follow 
up questions. 

Survey Analysis 

School survey data submitted electronically through the survey website were automatically 
stored in a downloadable database. Data from school surveys that were returned to us by mail 
were manually entered into the database. Once all of the survey information was compiled, we 
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examined the percentage of respondents by response option for each question (see Technical 
Appendix F). Given that our sample design stratified by school type, where appropriate, we 
tested for significant differences among elementary, middle, and high schools. As opposed to our 
case study sample which was selected with school performance in mind, our survey sample did 
not consider student outcomes as a primary stratifier. Nevertheless, we were still interested in 
exploring associations between the survey data and indicators of progress. To do so, we 
developed two metrics based on an estimate of average school performance using student-level 
CST scores in English Language Arts (ELA) and Math (controlling for student- and school-level 
characteristics; see Appendix C-3 for detail) and the API similar schools rank. Our first metric, 
defined high and low performers only using the CST measure. Our second metric was more 
restrictive and used both the CST performance measure and the similar schools rank. Although 
we tested for differences in survey results among high and low performers using both metrics, 
we did not find any revealing or consistent patterns. 

Eor each of the five open-ended survey questions, we developed an analysis framework by first 
creating a set of discrete categories which captured the most prevalent themes from the data. We 
then coded the incidence of these categories in each response. Synthesizing the data in this way 
allowed us to identify patterns and quantify those patterns across the surveyed schools. A similar 
framework was used for the open-ended district questions. We used the student achievement 
analysis results and prior research in the field of school accountability to inform our analyses. 

The following chapters present the results of the quantitative and qualitative data analyses. 
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Chapter 3: Impact of Program Participation 
on Student Achievement 



Introduction 

A major research question for this study and of significant policy interest is whether the HPSGP 
had an impact on student performance in participating schools. In this chapter, we examine 
statewide and school-level achievement trends associated with the implementation of the 
HPSGP. Given the program differences among HPSGP schools in addition to the fact that the 
program targeted the lowest performing schools, we first describe the selection of HPSGP and 
relevant comparison schools used for these analyses. We provide descriptive school-level 
performance trends of these 
schools according to state and 
federal accountability measures 
of performance. Using statewide 
student-level data and 
statistically controlling for key 
differences among the HPSGP 
and comparison schools, we 
explore the impact of the 
program on student achievement. 

These statewide results are 
complemented by analyses of a 
longitudinal dataset of a large 

school district in the state which, unlike state data, allows us to link the performance of 
individual students over time. 



Key Finding 

• On average, HP Only plus planning schools showed 
gains in student performance during the period of 
program implementation. 

• However - when statistically controlling for student 
and school characteristics - the differences in 
student performance between HP Only plus planning 
and comparison schools across a variety of 
comparative analyses were negligible. 



Steps Preliminary to the Analysis 

There were three important activities undertaken prior to analyzing the effect of participating in 
the HPSGP on student achievement are described below. First, given dual participation in other 
state programs, we identified a sub-group of HPSGP schools. Second, an appropriate comparison 
group of schools were selected against which HPSGP school student performance can be 
compared and contrasted. As described below, this activity was not as straightforward as might 
be expected. Last, we generated descriptive profiles of these two groups of schools to show how 
they appear in regard to student composition and student performance. 



The analyses present in this report exclude alternative, continuation, special education, state special, juvenile hall, 
community day, and adult education schools. Only those schools with a school type of elementary, middle/junior 
high, or high school in CBEDS are included. 
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Selecting the Subgroup of HPSGP Schools 

Given the variation in the timing of the funding and participation in other programs, 
the achievement analyses are limited to schools that did not participate in the ll/USP 
or CSR and received planning grant and the first round of implementation funds in 
the first year. 

As described in Chapter 1, HPSGP includes a number of schools that also participated in II/USP 
and/or CSR. Even among the HPSGP schools that did not participate in II/USP or CSR, there are 
differences in the timing of the first year implementation funds, with some schools receiving a 
second round of funding later. In addition, because the planning year was optional and due to the 
different selection criteria for implementation funds, not all of these schools received both 
planning and implementation funds. These achievement analyses focus on the effects of HPSGP 
in schools that were fully funded through the program (i.e., received a planning grant and first 
round of implementation funds ) and did not participate in II/USP and/or CSR. We refer to this 

7Q 

subset as the “HP Only plus planning” schools. 

Restricting the achievement analysis to HPSGP schools that received planning and first round 
implementation funds, and did not participate in II/USP and/or CSR, was an evaluation decision 
based on input from the CDE and the study’s Advisory Committee. The II/USP and CSR are no 
longer funded, and as future HPSGP funding will no longer be split between these three 
programs, the CDE and committee members believed that evaluation should focus on schools 
that did not also participate in II/USP and/or CSR. This group was further restricted to those that 
received a planning grant and first round of implementation funding. 

Essentially, the desire was to estimate an HPSGP effect that was not distorted by any previous 
similar interventions that targeted low performing schools in California. Even though it would 
have been possible to account for the influence of these other programs in a regression 
framework, the approach of limiting the HPSGP group as described above simplifies this. The 
trade-off is that the conclusions of this evaluation regarding the effects of the HPSGP on student 
achievement can only be generalized to schools that received both planning and one-time first 
year implementation funds, and that did not participate in lEUSP and/or CSR. In other words, the 
results do not answer the question of the impact of this program on HPSGP schools that had 
previously participated in lEUSP and/or CSR.^° 



HPSGP planning funds were allocated based on 2000 API Growth, and HPSGP implementation funds were 
allocated based on 2001 API Growth. 

By “first round of implementation funds,” we refer to the schools that received the first round of implementation 
funds during the course of the first year of the program (2002-03). As described in Chapter 1, 98 schools were 
selected later and received the first year of implementation funds at the end of the 2002-03 school year and 
beginning of the 2003-04 school year. 

The Year 1 report for this evaluation referred to these schools as “Pure -Pure” schools. To make the label more 
intuitive, we call them “HP Only plus planning” in this report. 

Earlier in the evaluation, we conducted achievement analyses that included schools that were jointly funded by the 
HPSGP and II/USP or CSR. Although the analysis approach was refined over time, these earlier analyses including 
all HPSGP schools showed similar results to those in the current chapter. 
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What are HP Only plus planning schools? 

For the purposes of this report, we grouped schools that received HPSGP 
implementation funds into three broad categories: 

• Mixed Program Schools: Schools that were jointly funded by HPSGP and ll/USP 
or CSR. Among the 658 schools receiving HPSGP implementation funds, 307 
(46.7%) were mixed program schoois. 

• HP Only: HPSGP schoois that received implementation funds and did not 
participate in ll/USP or CSR. Of the total schools receiving HPSGP implementation 
funds, 351 (53.3%) were HP Only. 

• HP Only plus planning: These schools are a subset of the HP Only group. These 
HPSGP schools a) did not participate in ll/USP or CSR, b) received a pianning 
grant, and c) received the first round of impiementation funds. Aii three of these 
criteria must be met. In addition, this group was limited to schoois with elementary, 
middle/junior high, and high school type in CBEDS. Of the total HPSGP schools, 
229 (34.8%) were HP Only plus planning. 

Please note that the above schools may be participating in other non-specified 
programs, such as the federal Program Improvement and Reading First. 



Selecting the Comparison Schools 

As HPSGP schools were purposely selected as the lowest performing in the state, a 
perfect comparison group of schools for these analyses does not exist. Consequent 
trade-offs were necessary regarding sample size and the need to keep comparison 
schools as similar to HPSGP schools as possible. 

Selecting the comparison schools for these analyses is critical to producing the most objective 
possible results in regard to the HPSGP impact. These schools are used for comparing academic 
achievement in schools participating in the HPSGP to what we see in similar schools over this 
same period. This provides the basis for estimating what academic performance would have been 
expected in the HP Only plus planning schools had they not been in the program. 

The ideal comparison group would exhibit the same characteristics as the HP Only plus planning 
schools with the only major difference being participation in the HPSGP. However, as the lowest 
performing schools in the state were targeted for this program, completely similar comparison 

o 1 

schools do not exist. Exhibit 3.1 shows the overall 2001 API Growth distribution of HP Only 
plus planning schools (the dotted line). It also shows the 2001 API Growth distribution of all 
schools in California that did not participate in any other similar state program, such as II/USP or 
CSR (the continuous line). As observed, HPSGP almost exclusively targeted the very lowest 
performing schools in the state, leaving only a small number of schools with similar 2001 
academic performance for the comparison group. 



See Chapter 1 for definition of API Growth and API Base. 
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Exhibit 3.1. 2001 API Growth Distribution of HP Only + Planning Schools and All 
Non-ll/USP and Non-CSR California Schools 




2001 API Growth 



* Note: Graphs excludes schools participating in CSR and/or ll/USP, as weli as alternative, continuation, 
special education, state special. Juvenile hall, community day, and adult education schools. 



Understanding the selection process used to assign schools to the program is crucial for 
determining appropriate comparison schools and to tease out possible effects of the HPSGP on 
student achievement. In the case of the HPSGP, the CDE invited all eligible schools to apply to 
the program as the first step of the selection process. In 2001, a total of 713 schools were invited 
to apply, out of which 646 submitted applications (see Exhibit 3.2). Implementation funds were 
then distributed to schools ranked from low to high according to their 2001 API Growth, until 
the funds were exhausted. Eater in October 2003, as additional funds became available, a new 
round of schools (n = 113) was invited to apply to the program. All of these schools applied, and 
all but 19 received HPSGP implementation funds. 
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Exhibit 3.2. The HPSGP Selection Process 

Note: Shows all schools that were Invited, applied, and received HPSGP implementation funds 





Originally Invited (2001) 


Late Invited (2003) 






Invited to 
Apply 


Applied 


Invited 
But Did 
Not 
Apply 


Invited to 
Apply 


Applied 


Invited 
But Did 
Not 
Apply 


Total 


Schools that received only 
HPSGP planning funds 


53 


53 




0 


0 






Schools that received only 
HPSGP implementation funds 


297 


297 




30 


30 






Schools that received planning 
& Implementation funds 


243 


243 




64 


64 






Sub-Total: 

Schools that received HPSGP 
implementation funds 


540 


540 




94 


94 




634* 


Schools that were not selected 
to participate In the HPSGP 


120 


53 


67 


19 


19 


0 




Total 


713 


646 


67 


113 


113 


0 





Source: Data provided by the California Department of Education. 

* This total is lower than the 658 that appears in the earlier textbox as it excludes 24 schools funded through the CSR program. 



As mentioned above, for those schools that were originally invited in 2001 and applied to the 
program, HPSGP implementation funds were allocated using the 2001 API Growth. Schools 
were ranked from lowest to highest API, and implementation funds were distributed with priority 
given to the lowest-scoring schools until funds were exhausted. Using the same method, we 
selected the lowest-performing schools (according to their 2001 API Growth) that were not 
chosen for the HPSGP. In addition, we ruled out schools that had participated in II/USP and/or 
CSR to ensure that the comparison schools do not reflect the influence of these other programs. 

In order to evaluate the impact of the HPSGP by school type, we require a large enough sample 
size of elementary, middle, and high schools. Ideally, this would include at least 30 schools in 
each of these school-level categories to provide enough statistical power to evaluate policy 
effects. At the same time, although having relatively large comparison groups is important, 
increasing the number of schools also has the effect of reaching further into the pool of higher- 
performing schools. With these trade-offs in mind, we selected a comparison group that included 
30 schools at each school level. 



Descriptive Differences of HP Oniy pius Pianning and Comparison Schoois 

Because a perfect comparison group for these analyses does not exist, comparison 
schools differ from HP Only plus planning schools in regard to student 
demographics and a number of student performance measures. 

Differences in API Growth 

Exhibit 3.3 presents the average 2001 API Growth of HP Only plus planning and comparison 
schools by school type. As can be observed, the HP Only plus planning schools are at the lower 
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end of the 2001 performance distribution. Their 2001 API Growth average is below those of the 
comparison groups for all three school types. This is especially noticeable for middle schools. 
Fewer comparison schools performed at the same level as HPSGP middle schools, which 
required us to include relatively higher-performing middle schools in order to reach the target 
sample size of 30 observations and which affects achievement comparability. 



Exhibit 3.3. Average 2001 API Growth of HP Only + Planning and Comparison 
Schools 



CBEDS School 
Type 


HP Only + Planning 
Schools 


Comparison Schools 


Total 




Number of 
Schools 


Average 
2001 API 
Growth 


Number of 
Schools 


Average 
2001 API 
Growth 


Number of 
Schools 


Average 
2001 API 
Growth 


Elementary Schools 


142 


472.3 


30 


486.5 


172 


475.0 


Middle Schools 


48 


460.0 


30 


526.4 


78 


485.5 


High Schools 


39 


469.6 


30 


499.2 


69 


482.9 


Total 


229 


469.0 


90 


504.1 


319 


479.5 



NOTE: This table includes the subset of HPSGP schools that did not participate in any other state reform program, and 
received planning grants and first round of implementation funds. We refer to this subset as “HP Only + Planning” schools. 



Differences in Student Characteristics 

Including higher-performing schools for the purpose of increasing the sample size of the 
comparison group has the unintended consequence of using schools with different demographic 
characteristics in relation to the HP Only plus planning schools. This is due to the fact that 
academic performance is highly correlated with socio-economic status (SES), ethnicity, and 
other demographic characteristics. Exhibit 3.4 shows that in 2001, HP Only plus planning 
schools had, on average, a higher percentage of students who are English learners, Hispanic, 
African-American, eligible for free or reduced price lunch, and with lower parental education. In 
2006, these differences remained almost identical (see Technical Appendix C-2). 
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Exhibit 3.4. Student Demographic Characteristics of HP Only + Planning, 
Comparison, and All Regular Public Schools in California, 2000-01 



100% 

90% 

80% 
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01 
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3 

+J 

U1 
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im 

01 
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20 % 
10 % 
0% 



Source: 2001 API Growth database, enrollment weighted estimates. Enrollment weighted estimated indicate average 
student characteristics (as opposed to average school characteristics provided by unweighted estimates). 



■ HP Only + Planning 



■ Comparison Schools 

■ State Averaoe 




Eligible for English Hispanic or African Parents Not Special 

Free or Learners Latino American High School Education 

Reduced Graduates Students 

Price Lunch 



Differences in Meeting API Growth Targets 

Before looking for relationships between HPSGP participation and student academic 
achievement, this section provides purely descriptive data showing the school-level performance 
trends of HP Only plus planning, comparison, and all schools statewide. 

In California’s accountability system, the main school-level performance indicator is the API. 
Schools with a schoolwide API lower than 800 are given an annual growth target that is five 
percent of the difference between 800 and the school’s current score. There are separate growth 
targets for numerically significant subgroups such as ethnic subgroups and socio-economically 
disadvantaged students. Exhibit 3.5 shows the percentage of HP Only plus planning and 
comparison schools that met their schoolwide growth targets from 1999-2000 through 2005-06. 
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Exhibit 3.5 Percentage of HP Only + Planning and Comparison Schools Meeting 
Schoolwide API Growth Targets, 1999-2000 to 2005-06 




Year 



The exhibit shows that a higher proportion of HP Only plus planning than comparison schools 
reached their schoolwide API growth targets in the school years 1999-2000 through 2003-04. For 
instance, in the 2001-02 school year, 63 percent of the HPSGP schools reached their schoolwide 
growth targets compared to 58 percent of the comparison schools. Only during the last two 
school years analyzed did a higher proportion of comparison schools meet the schoolwide API 

on 

growth target. 

However, these results cannot be interpreted as evidence of the HPSGP’ s success for several 
reasons. First, greater percentages of HP Only plus planning schools than comparison schools 
were meeting their schoolwide targets before the program was actually implemented. Second, 
this analysis does not control for student- or school-level characteristics. And finally, given that 
HPSGP schools are farther down the performance spectrum, they have more room for 
improvement in terms of their API performance. Due to the manner in which the API is 
calculated, which results in more points gained when students in the lower levels of proficiency 
improve their achievement, these schools can potentially reach their growth targets more 

O'! 

easily. For these reasons, this type of analysis should only be considered descriptive. 



It is interesting to note that the school year 2002-03 was particularly successful in terms of schools meeting their 
schoolwide API targets. That year, the Norm Reference Test (NRT) was changed from the SAT-9 to the CAT/6. 

Please refer to http://www.cde.ca.gov/ta/ac/ap/documents/infoguide05b.pdf for further information on this topic. 
API growth targets are five percent of the difference between 800 points and their current API, whenever this 
number is smaller than 800. This means that smaller APIs generate larger growth targets because they are farther 
away of the overall long-term performance target of 800 points. 
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Differences in Statewide Deciie Ranks 

Another analysis using school-level API measures is the movement of HPSGP and comparison 
schools across state ranks over the years. As shown in Exhibit 3.6, 95 percent of HP Only plus 
planning schools were in the lowest decile in the 2000-01 school year. On the other hand, only 
about half of the comparison schools were in the lowest decile that year. Neither group had any 
schools above decile 2 in the school year 2000-01. Over time, some schools in both groups have 
moved out of the lowest state rank deciles. In fact, one HPSGP school reached the state decile 
rank 9 in 2005-06.^^ 



Exhibit 3.6. API Statewide Decile Ranks for HP Only + Planning and Comparison 
Schools, 1999-2000 to 2005-06 


YEAR 


Decile 1 


Decile 2 


Decile 3 


Decile 4 


Decile 5 


Decile 6 


Decile 7 Deciles Decile 9 Decile 10 


HP Only + 


Planning (N = 


229) 












1999-00 


74% 


11% 


0.4% 










2000-01 


95% 


3% 












2001-02 


78% 


18% 


0.4% 










2002-03 


67% 


24% 


6% 


0.4% 








2003-04 


56% 


32% 


9% 


1% 


0.4% 






2004-05 


53% 


31% 


10% 


3% 


1% 


0.4% 


0.4% 


2005-06 


49% 


31% 


11% 


5% 


2% 




0.4% 0.4% 


Comparison Schools (N 


= 90) 












1999-00 


12% 


47% 


19% 


1% 








2000-01 


41% 


58% 












2001-02 


32% 


43% 


18% 


3% 








2002-03 


29% 


38% 


17% 


4% 


2% 






2003-04 


32% 


33% 


18% 


8% 


1% 


1% 




2004-05 


31% 


32% 


20% 


8% 


1% 


1% 


1% 


2005-06 


28% 


29% 


19% 


14% 


1% 


1% 





Note: Schools participating in the Alternative School Accountability Model (ASAM) or with missing state rank are not 
included. 



Differences in Meeting Annuai AYP Targets 

In addition to the state accountability system, schools in California also have to comply with the 
federal NCLB requirements for Adequate Yearly Progress (AYP). Exhibit 3.7 shows the percent 
of HP Only plus planning and comparison schools that have been able to meet AYP. As 
observed, a consistently lower percentage of HP Only plus planning schools met the AYP targets 
across all years in relation to the comparison group. This is somewhat contrary to the API results, 
in which HP Only plus planning schools outperformed the comparison schools before 2004-05 
(see Exibit 3.5). It is relatively easier for lower performing schools (in this case, the HP Only 
plus planning schools) to show progress on the API, given that it is based on growth, as opposed 
to the AYP, which is based on performance levels. Eurthermore, the percentage of schools 
meeting all AYP targets has increased more in the comparison than the HP Only plus planning 
schools. Between 2002-03 and 2005-06, this figure has increased by nearly 12 percent in the 
comparison group, notably larger than the 7 percent of HP Only plus planning schools. 



Note that the API was not designed to generate longitudinal analyses over time. Its composition in terms of tests, 
subjects and weights changes from year to year, and therefore is not comparable across years. 

Even though these schools have made considerable progress over these years in terms of academic performance, 
only one school reached the 800 points threshold on its API Base between 2001 and 2006. This was an HPSGP 
school that reached decile 9 in 2006, with an API Base of 856. 
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Exhibit 3.7. Percentage of HP Only + Planning and Comparison Schools Meeting 
All AYP Targets, 2002-03 to 2005-06 




Year 



Differences in Percentage of Students Proficient or Above in ELA and Math 

To supplement these school-level analyses, we also examined trends in the percentage of 
students ranked as proficient or advanced in the CST English language arts (ELA) and math tests 
(see Exhibits 3.8 and 3.9, respectively).^*’ HP Only plus planning schools, on average, showed 
gains in the percentage of students scoring proficient or advanced on the CST. The percentage 
increased from 9 percent to 24 percent in English language arts and from 12 percent to 29 
percent in math from 2001-02 to 2005-06. The averages for comparison schools and all other 
schools showed similar trajectories. 

With respect to differences between the school groups, HP Only plus planning schools had a 
consistently lower percentage of students performing at proficient or advanced in relation to 
comparison schools. On the other hand, although the percentages for math are remarkably 
similar over time, during the last year HP Only plus planning schools have been able to improve 
in terms of this performance measure more than comparison schools. 



Please see Technical Appendix C-5 for these results for student subgroups. 
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Exhibit 3.8. Percentage of Students in HP Only + Planning and Comparison 
Schools Classified as Proficient or Advanced in CST ELA, 2001-02 to 2005-06 




Exhibit 3.9. Percentage of Students in HP Only + Planning and Comparison 
Schools Classified as Proficient or Advanced in CST Math, 2001-02 to 2005-06 




Year 



Even though these descriptive figures provide important school-level information about the 
progress of the HP Only plus planning and comparison schools on state and federal 
accountability measures, they cannot be used to deduce the impact of the HPSGP on student 
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achievement, as they do not control for differences in student- and school-level characteristics 
that may also influence academic achievement. 



Analysis of the Impact of Program Participation 

No substantial differences in student performance between HP Only plus planning 
and comparison schools were found. 

Improving student achievement is central to the purpose of the HPSGP and to this evaluation. In 
attempting to measure the impact of the program on this dimension, we must take into account 
the demographic characteristics of the student population and the relative starting points in 
regard to student achievement of HP Only plus planning and the comparison schools. 

These analyses need to be based on performanee trends over time, rather than any performanee 
gap in a particular year. Accordingly, we use statewide student-level STAR data from 1997-98 
through 2005-06. These data capture all students served in public schools in grades 2 through 1 1 
(except for the small percentage of students exempted from the STAR test). These state data 
allow us to statistically control for student-level demographic characteristics, such as gender, 
ethnicity, and eligibility for free or reduced price lunch, but do not contain the information 
needed to link individual student performance over time. 

We used regression analyses to statistically control for differences in student- and school-level 
background characteristics across the HP Only plus planning and comparison schools. Also to 
account for the fact that we are combining student- and school-level variables in these equations, 
we use a Hierarchical Linear Model (HLM) framework. Technical Appendix C-3 shows the 
student- and sehool-level demographic characteristics included as controls in the HLM 
regressions for this study. 

Determining Achievement Outcome Measures 

Evaluating the academic performance trends of HPSGP and comparison schools requires 
consistent student-level measures of performance over time. From the state’s STAR database, 
which provides student-level achievement data for the years 1997-98 through 2005-06, we used 
multiple measures to evaluate progress in academic achievement over this time period. The two 
types of statewide tests included in STAR during this period were Norm Refereneed Tests 
(NRT), including the SAT-9 and the CAT/6, and the standards-based CST. 

In regard to the NRT elements of STAR, the SAT-9 test was administered from 1997-98 through 
2001-02, and the CAT/6 from 2002-03 through 2003-04 for all grades 2 through 11. However, 
starting in 2004-05, this test was only implemented in grades 3 and 7. Therefore, we were not 



The literature that links family characteristics, such as ethnicity and parental education, to student academic 
achievement is extensive, dating back to the Coleman Report (Coleman, Campbell, Hobson, McPartland, Mood, 
Weinfeld, & York, 1966). Peers and other school-level factors also influence the educational experience of students, 
bringing a social aspect into the equation and generating a very complex system of interacting factors at different 
levels. Hanushek (2002) provides an overview of the school-level factors that affect academic achievement. 
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able to include CAT/6 data beyond 2003-04 given that the grades do not align. The standards- 
based CST is available for grades 2 through 11 from 2001-02 through 2005-06. 

Because of these changes in the tests administered, it was necessary to standardize the results; 
otherwise the scores from these three tests cannot be compared over time. In addition, CST 
scores must be standardized within grade levels because the test is not “vertically equated,” 
meaning that CST scale scores across grades are not comparable. For example, a CST scale score 
of 450 points in grade 3 is not necessarily a better score than 400 points in grade 2. 

Once standardized, the resulting scores indicate how far away (in standard deviations)^^ each 
student is from the state average in a specific subject. The subjects included in this analysis are 
language, mathematics, and reading for the SAT-9 and CAT/6, and English language arts (ELA) 
and mathematics for the CST. Although standardization allows us to make valid comparisons 
against a specific benchmark (i.e., the average performance of the state), it does not permit 
measurement of absolute growth since by standardizing we are resetting the average performance 
of the state to zero each year. 



Results from the Statewide Analysis 

The analysis presented in this section is designed to capture the effect of participation in a fully 
implemented HPSGP^^ by comparing average test scores of HP Only plus planning and 
comparison schools when student- and school-level characteristics are held statistically constant. 
This statewide analysis section first presents findings across all students, and then findings by 
individual sub-groups of students. Because the purpose is to focus on achievement trajectories,'^^ 
the model incorporates a time dimension. Eor a detailed discussion of the equations used, please 
refer to Technical Appendix C-4. 

In these regressions, we analyze whether students in HP Only plus planning schools were able to 
improve their academic achievement over time at higher rates than those in comparison schools. 
Our variables of interest are the interaction terms of time and HPSGP. These interactions pick up 
differences in the growth rates of student academic achievement between HP Only plus planning 
and comparison schools. 

The numbers presented in the following tables represent the HEM regression coefficients of 
these interaction terms of time and participation in HPSGP. Given that the outcome variable is in 
standard deviations, they indicate the difference in the academic growth rate of students in the 
HP Only plus planning versus comparison schools in terms of standard deviations in the NRT 
and CST. 



We standardize scale scores within grades to a mean of zero and a standard deviation of one. A standard deviation 
is a statistical measure of how data are dispersed around the average or mean value, based on a normal distribution 
where about two thirds of the values lie within one standard deviation from the mean. It is a useful way to compare 
differences between two groups with different ranges of values. 

By “fully implemented,” we mean schools that received HPSGP planning grants and the first round of 
implementation funds. 

As mentioned above, these trajectories indicate how far students in HP Only plus planning and comparison 
schools are from the state average in a specific subject. 
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Is “statistically significant” the same as “educationaiiy significant”? 

To help assist readers in interpreting the resuits in the foiiowing sections, we refer to educationai 
iiterature that considers effect sizes smaiier than 0.25 standard deviations as “educationaiiy non- 
significant.” 

Cohen (1969) proposes three effect size categories: 0.25 standard deviations or less as “small,” 
0.25 to 0.4 as “medium,” and 0.4 or more as “large.” This gives us an important reference point 
in distinguishing between “statistical significance” and “educational significance.” In other words, 
while two results may be considered different in a statistical sense, the magnitude of the 
difference may be so small that it is considered as largely irrelevant from a policy perspective. 



The following exhibits refer to the 2005-06 school year as the fourth year of implementation, 
which was based on the schools’ performance during the first three years of the program. While 
most of the HP Only plus planning schools were provided a fourth year of funding, 33 of these 
schools were subject to SAIT in 2005-06 and therefore were no longer part of the program. 
However, given the long HPSGP involvement of these schools we kept them in the HP Only plus 
planning group in 2005-06. 

Overall Statewide Results 

Exhibit 3.10 presents the estimated effect of program participation on annual growth rates of 
academic achievement in HP Only plus planning elementary schools.^' All three NRT subjects 
(reading, math, and language arts) show higher achievement improvement in elementary HPSGP 
schools in the year before the implementation of the program (i.e., in 2000-01). While we only 
observe a statistically significant (but negative) HPSGP effect during the planning year in 
language, the NRT scores indicate a very small positive HPSGP implementation effect across all 
three subjects during the first implementation year. During the second implementation year, HP 
Only plus planning schools again outperformed comparison schools in reading and math. 
Unfortunately, there are no NRT scores available for all grades after 2004, so we supplemented 
this analysis with an evaluation of the CST performance trends. 



Technical Appendix C-5 presents the complete statewide HLM regression results. 
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Exhibit 3.10. HLM Regression Results for HP Only + Planning and Comparison 
Elementary Schools 



Year 


NRT 

Reading 


NRT Math 


NRT 

Language 


CST ELA 


CST Math 


Year 1998-99 


- 0.005 


0.01 


- 0.02 




Year 1999-00 


0.02 


- 0.02 


0.03** 






Year 2000-01 


0.07*** 


0.04*** 


0.07*** 






Year 2001-02 (Planning) 


0.004 


- 0.003 


- 0 . 03 ** 




Year 2002-03 (Implementation 1) 


0.05*** 


0.03*** 


0.06*** 


0.04*** 


0.06*** 


Year 2003-04 (Implementation 2) 


0.03** 


0.04*** 


0.02 


0.02* 


0.04*** 


Year 2004-05 (Implementation 3) 




- 0 . 05 *** 


- 0 . 06 *** 


Year 2005-06 (Implementation 4) 




- 0 . 03 ** 


- 0 . 03 ** 



Note: These coefficients indicate the effect of the HPSGP on student achievement in standard deviations. Shaded 
cells Indicate that tests were not administered In all grades those years. 

* indicates statistical significance at 10%, ** at 5%, and *** at 1%. 



The results of the CST regressions point in the same direction as those of the NRT in the first 
two implementation years. However, HP Only plus planning elementary schools experience a 
slower growth rate of academic performance in relation to comparison schools in the third and 
fourth years of implementation, which raises questions about the sustainability of the previously 
observed positive program effect. 

Perhaps the most important observation to take away from this exhibit is that while many of 
these results are statistically significant, they are so small that we must be careful not to associate 
too much meaning to the distinctions between the positive and negative relationships observed 
above. 

Exhibit 3.11 presents the HLM results for middle schools, which indicate that HP Only plus 
planning schools fared less well than comparison schools in improving student academic 
performance during the planning year, followed by a generally positive program effect during the 
first two implementation years. Middle school CST scores are consistent with NRT scores during 
the two years they overlap, but then fall off to more mixed results during the third and fourth 
implementation years. As with the elementary school findings, however, the effect sizes are so 
small that it is not clear that any of the differences seen above (negative or positive) are very 
important from the perspective of meaningful educational growth. 
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Exhibit 3.11. HLM Regression Results for HP Only + Planning and Comparison Middle 
Schools 



Year 


NRT 

Reading 


NRT Math 


NRT 

Language 


CST ELA 


CST Math 


Year 1998-99 


- 0 . 04 *** 


- 0 . 06 *** 


- 0 . 05 *** 




Year 1999-00 


- 0.005 


- 0.008 


0.004 






Year 2000-01 


0.009 


- 0.003 


0.008 






Year 2001-02 (Planning) 


- 0 . 04 *** 


- 0 . 02 *** 


- 0 . 02 *** 




Year 2002-03 (Implementation 1) 


0.05*** 


0.01* 


0.02*** 


0.04*** 


0.01* 


Year 2003-04 (Implementation 2) 


0.06*** 


0.006 


0.05*** 


0.01* 


0.00007 


Year 2004-05 (Implementation 3) 




0.0006 


0.01* 


Year 2005-06 (Implementation 4) 




- 0 . 02 *** 


- 0.01 



Note: These coefficients indicate the effect of the HPSGP on student achievement in standard deviations. Shaded 
cells indicate that tests were not administered in all grades those years. 

* indicates statistical significance at 10%, ** at 5%, and *** at 1%. 



The HLM results for high sehools, as shown in Exhibit 3.12, show similarities with the prior two 
sets of results. What sets high sehools apart, however, is that comparison schools outperformed 
HP Only plus planning schools during the second implementation year. There is no consistent 
positive trend over time in HPSGP high schools, and the data show an alternating pattern of 
positive and negative effects from year to year. Again, the effect sizes are very small, and one 
should not place too much importance on yearly swings. Given the size of the effects and the 
observed alternating pattern, the soundest interpretation may be an overall pattern of no effect, 
with some slight variations. 



Exhibit 3.12. HLM Regression Results for HP Only + Planning and Comparison High 
Schools 



Year 


NRT 

Reading 


NRT Math 


NRT 

Language 


CST ELA 


CST Math 


Year 1998-99 


0.02*** 


- 0 . 01 * 


- 0.004 




Year 1999-00 


0.01* 


0.02** 


0.02*** 




Year 2000-01 


0.007 


0.008 


0.02*** 




Year 2001-02 (Planning) 


- 0 . 03 *** 


- 0 . 02 *** 


- 0 . 04 *** 




Year 2002-03 (Implementation 1) 


0.07*** 


0.01 


0.08*** 


0.03*** 


0.02*** 


Year 2003-04 (Implementation 2) 


- 0 . 04 *** 


- 0 . 02 *** 


- 0 . 05 *** 


- 0 . 02 ** 


- 0 . 01 * 


Year 2004-05 (Implementation 3 




0.02** 


0.02*** 


Year 2005-06 (Implementation 4) 




- 0 . 04 *** 


- 0 . 04 *** 



Note: These coefficients indicate the effect of the HPSGP on student achievement in standard deviations. Shaded 
cells indicate that tests were not administered in all grades those years. 

* indicates statistical significance at 10%, ** at 5%, and *** at 1%. 



As shown in Exhibit 3.13, with these annual differences between HP Only plus planning and 
comparison schools, we can estimate the cumulative impact of the HPSGP by summing the 
statistically significant differences across all implementation years by GST subject test. Eor 
example, for elementary schools, this reveals an overall HPSGP effect of -0.02 and 0.01 standard 
deviations annually in EEA and math, respectively. In other words, HPSGP elementary schools 
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have improved less in English language arts over time than comparison schools after the 
introduction of the HPSGP^ and in math the reverse is the case. In the case of middle schools, the 
estimated total effects are equal to 0.03 and 0.02 standard deviations in ELA and math, 
respectively. Einally, HPSGP high schools were outperformed in both subjects by comparison 

42 

schools after the implementation of the program (by 0.01 standard deviations). 

Exhibit 3.13. Cumulative Impact of the HPSGP in HP Only + Planning Schools 





CST ELA 


CST Math 


Statewide Elementary HP Only + Planning 


-0.02 


0.01 


Statewide Middle HP Only + Planning 


0.03 


0.02 


Statewide High HP Only + Planning 


-0.01 


-0.01 



Note: These figures are estimated summing all the statistically significant coefficients of Exhibits 3.10, 3.1 1 , and 3.12. 

As noted above, these effect sizes are arguably too small to translate into meaningful differences 
in terms of educational benefit between HP Only plus planning and comparison schools. Given 
the very small size of these differences, these data seem to show that HP Only plus planning and 
comparison schools performed virtually the same over the period of HPGSP implementation 
when controlling for student demographic characteristics. It is important not to over-emphasize 
statistical significance whenever effect sizes are close to zero. This is especially relevant given 
the large sample size at our disposal for this evaluation. The larger the sample, the higher the 
power to reject any null hypothesis, and therefore it is easier to find statistically significant 
differences. 

Statewide Student Sub-Group Results 

Despite the general conclusion of no meaningful difference between HP Only plus planning and 
comparison schools above, it is possible that the program had more substantial effects for certain 
subpopulation of students. Analyzing test score trajectories for all students in HP Only plus 
planning and comparison schools may mask differences across important sub-populations of 
students. Therefore, in this section we analyze the impact of the HPSGP on students who are 
English learners, Hispanic, African-American, Asian, in special education, or eligible for free or 
reduced price lunch. The analyses by subpopulations focus only on the CST results in order to 
cover the whole implementation period of the HPSGP. 

Exhibit 3.14 presents these results. Some general patterns are observed in the elementary and 
middle schools, where statistically significant differences tend to be positive in the first two 
years of the program (2002-03 and 2003-04), and negative thereafter. In other words, certain 
types of students in HP Only plus planning schools tend to outperform comparison schools 
during the first two years of the program and then fall behind the comparison group in the later 
years. 



The following is an example of what an annual difference of 0.01 standard deviations means in more concrete 
terms. Second graders in elementary HP Only plus planning and comparison schools had an average scale score of 
334.5 with a standard deviation of 82.6 points on the CST in 2006 (CST math scale scores). A difference of a 0.01 
standard deviation means, on average, that second grade students in HP Only plus planning elementary schools had 
scale scores that were about 0.8 points (82.6 * 0.01) higher than those enrolled at comparison sites. 
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Note that this general pattern is not a rule. An exception to this are English learners in HP Only 
plus planning elementary schools, who demonstrated higher rates of achievement than those in 
the comparison group by 0.07 and 0.10 standard deviations in ELA and math, respectively, in the 
fourth year of the program (2005-06). Similarly, African-American students in HPSGP 
elementary schools and special education students in HPSGP middle schools demonstrated larger 
achievement gains than their peers in comparison schools in the last year of the program. 

Exhibit 3.14. HLM Regression Results of the HPSGP Effect on Certain Student Groups in 
HP Only + Planning Schools 



Student Group 


Year 


Elementary 

CST ELA CST Math 


Middle 

CST ELA CST Math 


CST ELA 


High 

CST Math 




Year 2002-03 


0.02 


0.04*** 


0.06*** 


0.04*** 


0.08*** 


0.02* 


English 


Year 2003-04 


- 0.002 


0.01 


0.03*** 


0.003 


- 0 . 04 *** 


- 0 . 03 ** 


Learners 


Year 2004-05 


- 0 . 06 *** 


- 0 . 06 *** 


- 0 . 05 *** 


- 0 . 02 ** 


0.0001 


- 0.0009 




Year 2005-06 


0.07** 


0.10*** 


- 0.01 


- 0 . 02 * 


- 0 . 06 *** 


- 0 . 04 *** 


Low Socio- 
Economic 


Year 2002-03 


0.03*** 


0.06*** 


0.03*** 


0.01 


0.04*** 


0.03*** 


Year 2003-04 


0.02 


0.04*** 


0.01 


0.0003 


- 0 . 04 *** 


- 0 . 04 *** 


Status 


Year 2004-05 


- 0 . 06 *** 


- 0 . 07 *** 


- 0.007 


0.006 


0.02** 


0.02*** 


Students^ 


Year 2005-06 


- 0 . 03 ** 


- 0.02 


- 0 . 02 ** 


- 0.01 


- 0 . 03 *** 


- 0 . 02 ** 




Year 2002-03 


0.04*** 


0.07*** 


0.04*** 


0.01* 


0.03*** 


0.02** 


Hispanic 


Year 2003-04 


0.009 


0.03* 


0.01 


- 0.001 


- 0 . 02 *** 


- 0 . 02 *** 


Students 


Year 2004-05 


- 0 . 05 *** 


- 0 . 05 *** 


- 0.01 


0.008 


0.02** 


0.02*** 




Year 2005-06 


- 0 . 04 *** 


- 0 . 05 *** 


- 0 . 01 * 


- 0.004 


- 0 . 04 *** 


- 0 . 03 *** 




Year 2002-03 


0.05 


0.002 


0.006 


- 0.02 


0.0008 


0.04 


African- 


Year 2003-04 


0.04 


0.06 


0.03 


0.05*** 


- 0 . 06 *** 


- 0.002 


American 

Students 


Year 2004-05 


- 0 . 09 ** 


- 0 . 16 *** 


0.04 


- 0.004 


0.03 


0.03 




Year 2005-06 


0.12*** 


0.19*** 


- 0.02 


- 0.01 


- 0 . 05 ** 


- 0 . 08 *** 




Year 2002-03 


- 0.09 


- 0.05 


0.03 


- 0.01 


0.02 


- 0.04 


Asian 


Year 2003-04 


0.25** 


0.28** 


- 0.006 


- 0.01 


0.08** 


0.02 


Students 


Year 2004-05 


0.12 


- 0.02 


- 0.002 


- 0.003 


0.03 


- 0.03 




Year 2005-06 


-0.05 


0.05 


- 0 . 20 *** 


- 0 . 13 *** 


- 0 . 16 *** 


- 0.03 




Year 2002-03 


0.003 


- 0.002 


- 0.02 


- 0.03 


0.007 


0.01 


Special 


Year 2003-04 


0.01 


0.02 


0.01 


0.0006 


- 0.02 


0.02 


Education 

Students 


Year 2004-05 


- 0 . 10 ** 


- 0 . 11 *** 


- 0 . 07 *** 


- 0 . 04 ** 


0.11*** 


0.11*** 




Year 2005-06 


0.01 


0.03 


0.07*** 


0.06*** 


- 0 . 08 *** 


- 0 . 09 *** 



Note: These coefficients indicate the effect of the HPSGP on student achievement in standard deviations. 
^ As measured by student’s eligibility for free or reduced price lunch. 

* indicates statistical significance at 10%, ** at 5%, and **’* at 1%. 



Exhibit 3.15 shows the cumulative program effect, summing all statistically significant 
differences in performance gains for student sub-groups between HP Only plus planning and 
comparison schools over the implementation years as was done for the earlier schoolwide 
analyses. Some of the sub-groups present the same program effects as the overall school-level 
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analyses with very small effect sizes (both positive and negative). Asian students show the 
largest effect sizes, but this tends to be for only one out the four years of implementation and 
positive for elementary students and negative for Asian students in middle and high school. 

Exhibit 3.15. Cumulative Impact of the HPSGP on Certain Student Groups in HP Only + 
Planning Schools 





Elementary 

GST ELA GST Math 


Middle 

GST ELA GST Math 


High 

GST ELA GST Math 


English Learners 


0.01 


0.08 


0.04 


0.00 


-0.02 


-0.05 


Lew Seclo-Ecenomic 
Status Students’* 


-0.06 


0.03 


0.01 


0.00 


-0.01 


-0.01 


Hispanic Students 


-0.05 


0.00 


0.03 


0.01 


-0.01 


-0.01 


African-American Students 


0.03 


0.03 


0.00 


0.05 


-0.11 


-0.08 


Asian Students 


0.25 


0.28 


-0.20 


-0.13 


-0.08 


0.00 


Special Education Students 


-0.10 


-0.11 


0.00 


0.02 


0.03 


0.02 



* As measured by student’s eligibility for free and reduced price lunch. 

Note: These figures are estimated summing all the statistically significant coefficients of Exhibit 3.14. 



In summary, while it is possible that the overall finding of no meaningful program effect found 
in the aggregate statewide analyses masks possible effects on important subpopulations of 
students, the results above show this not to be the case. Similar to the general pattern observed 
for all students, the sub-group analyses largely show fluctuations across grade levels and subjects 
between a slightly positive and a slightly negative HPSGP effect. The most defensible 
conclusion across these state-level sub-population analyses is no meaningful program effect, 
either positive or negative. 
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Results of Student-Level Longitudinal Analysis of Data from a Large Urban 
School District 

The statewide achievement analyses presented in the previous sections of this chapter analyzed 
academic performance of HP Only plus planning and comparison schools over time, based on 
their student population in a given year. Given that no unique student identifiers are available in 
the student-level STAR datasets, it is not possible to match individual students across years and 
track their achievement over time. Therefore, statewide analyses rely on longitudinal trends at 
the school-level to make inferences about the effects of the HPSGP. 

While the results of statewide analyses must clearly dominate this evaluation of the HPSGP, the 
fact that these statewide data do not feature student data with identifying information to allow the 
performance of individual students to be tracked over time poses some important data 
limitations. For example, student mobility can be important and is likely to occur more in low 
performing schools than in others. When tracking the performance of individual schools over 
time, as we are doing in the statewide analysis above, there is no way to know the extent to 
which we are tracking the impact of the HPSGP on students who have been present during the 
full four years of program implementation at the school as opposed to a constantly changing 
student population who would have much less opportunity to be affected by the HPSGP. To the 
extent that student mobility is greater at HPSGP than at comparison schools, we are comparing 
the results of relative mobility as opposed to relative stability combined with possible HPSGP 
impact. 

Thus, through the analyses presented in this section, we attempt to further investigate the 
question of HPSGP impact through the use of a longitudinally-linked student-level dataset from 
a large urban school district. These data include all students in the district across seven years 
(1999-2000 through 2005-06) with unique identifiers that allow us to follow the performance of 
individual students over time on the NRT reading, math, and language and CST ELA and math. 
The dataset contains a wealth of information on each student, including gender, primary 
ethnicity, English proficiency, parental education, eligibility for free or reduced price lunch, 
special education status, and current grade. 

While a relatively high number of HPSGP schools are located in this district, it is important to 
highlight that this district analysis does not replace, but rather complements, the statewide 
analyses in the previous sections, as the results cannot be generalized beyond the district. While 
it does not provide answers to our evaluation questions that can be extrapolated to the state as a 
whole, it may provide some insight into the question of the potential impact of not having 
individual linked student data at the state level on the statewide findings presented in this report. 

Given that students often move among schools over the years, we limited this analysis to those 
students who either have always or who never attended an HP Only plus planning school. This 
eliminates the complications raised by students who transfer between HPSGP and comparison 



AIR was requested not to identify the district. 
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schools during our period of analysis. In addition, we hold the sample of students constant over 
time from 1999-2000 through 2005-06.^"^ 

Students who always attended an HP Only plus planning school in the district for this time 
period constituted the HPSGP group for this analysis. Even though this may reduce the sample 
size somewhat, it is important to keep the sample of students constant over time, in order to 
compare academic performance trajectories before and after the program was put in place, and 
make sure that changes in academic performance are not due to changes in the student 
population attending these schools, as described above. 

In order to select a comparison group for these students, we no longer need to select entire 
schools, but rather individual students in the same school district. We matched each student in 
the HPSGP group with another student with similar initial academic achievement level (i.e., 
NRT reading in 1999-2000), primary ethnicity, eligibility for free or reduced price lunch, special 
education status, and grade from a non-HPSGP school. To supplement this matching, we 
selected these comparison students in schools with a similar percentage of students eligible for 
free or reduced price lunch as the school attended by the matched HPSGP student in 1999-2000. 
In other words, for each student in the HPSGP group, we found his/her most similar counterpart 
in the district at a non-HPSGP school based on these characteristics. The only difference 
between the matched pair is that one attended an HP Only plus planning school from 1999-2000 
through 2005-06, and the other did not (neither did the matched student attend II/USP or CSR 
schools). 

Overall District Data Results 

Exhibit 3.16 presents the results of the student- level longitudinal HEM regressions using district 
data."^^ The NRT scores in reading show a positive HPSGP effect of 0.07 standard deviations in 
reading during the first year of program implementation in 2002-03. However, the results 
following this first year of implementation suggest that students in comparison schools slightly 
outperformed students in the HPSGP schools. More specifically, there was a negative program 
effect of -0.05 standard deviations on NRT language in 2003-04; -0.04 standard deviations on 
CST EEA in 2004-05 and 2005-06; and -0.08 standard deviations on CST math in 2005-06. 



Following a consistent group of students over time is referred to as a balanced panel. 

Students in HPSGP and comparison schools did not have to attend the same school from 1999-2000 through 
2005-06. The only restriction is that students in the HPSGP group needed to always attend an HP Only plus 
planning school throughout the period, but they may have transferred from one to another. The same is true for 
students in the comparison group; they could move between schools, but always from a non-HP, non-II/USP and 
non-CSR school to another. 

Technical Appendix C-6 presents the complete HLM regression results. 
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Exhibit 3.16. HLM Regression Results for Students in HP Only + Planning and 
Comparison Schools in a Large Urban District 



Year 


NRT 

Reading 


NRT Math 


NRT 

Language 


CST ELA 


CST Math 


Year 2000-01 


- 0 . 07 *** 


-0.03 


- 0 . 10 *** 




Year 2001-02 (Planning) 


-0.02 


0.02 


0.09*** 




Year 2002-03 (Implementation 1) 


0.07*** 


-0.03 


0.01 


- 0 . 04 ** 


- 0 . 05 * 


Year 2003-04 (Implementation 2) 


-0.01 


0.03 


- 0 . 05 ** 


0.004 


0.02 


Year 2004-05 (Implementation 3) 




- 0 . 04 ** 


-0.002 


Year 2005-06 (Implementation 4) 




- 0 . 04 * 


- 0 . 08 *** 



Note 1 : These coefficients indicate the effect of the HPSGP on student achievement in standard deviations. Shaded 
cells indicate that tests were not administered in all grades those years. 

Note 2: An average of 23,540 observations per subject test were included in this analysis. 

* indicates statistical significance at 10%, ** at 5%, and *** at 1%. 



District Student Sub-Group Resuits 

We also analyzed performance trends by student sub-groups in this distrct, namely Hispanic and 
African-American students, students with low socio-economic status, and English learners.^’ 
Exhibits 3.17 and 3.18 present the HEM results for students eligible for free or reduced price 
lunch and Hispanic students, respectively. Given that 93 percent of the students in both the 
HPSGP and comparison schools are eligible for free or reduced price lunch, and 88 percent of 
them are of Hispanic origin, this extremely high overlap generates very consistent results with 
the overall findings for the district above. 



Exhibit 3.17. HLM Regression Results for Students Eligible for Free or Reduced Price 
Lunch in HP Only + Planning and Comparison Schools in a Large Urban District 



Year 


NRT 

Reading 


NRT Math 


NRT 

Language 


CST ELA 


CST Math 


Year 2000-01 


- 0 . 07 *** 


- 0 . 04 * 


- 0 . 11 *** 




Year 2001-02 (Planning) 


-0.01 


0.02 


0.10*** 




Year 2002-03 (Implementation 1) 


0.06** 


-0.03 


0.004 


- 0 . 04 * 


- 0 . 06 ** 


Year 2003-04 (Implementation 2) 


-0.004 


0.03 


- 0 . 05 * 


0.01 


0.04 


Year 2004-05 (Implementation 3) 




- 0 . 04 * 


0.004 


Year 2005-06 (Implementation 4) 




- 0 . 04 ** 


- 0 . 09 *** 



Note 1 : These coefficients indicate the effect of the HPSGP on student achievement in standard deviations. Shaded 
cells indicate that tests were not administered in all grades those years. 

* indicates statistical significance at 10%, ** at 5%, and *** at 1%. 



Asian and special education students were not included in these analyses due to small sample size. 
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Exhibit 3.18. HLM Regression Results for Hispanic Students in HP Only + Planning and 
Comparison Schools in a Large Urban District 



Year 


NRT 

Reading 


NRT Math 


NRT 

Language 


CST ELA 


CST Math 


Year 2000-01 


- 0 . 08 *** 


- 0 . 04 * 


- 0 . 10 *** 




Year 2001-02 (Planning) 


-0.01 


0.03 


0.10*** 




Year 2002-03 (Implementation 1) 


0.06** 


-0.02 


-0.002 


- 0 . 04 * 


- 0 . 06 ** 


Year 2003-04 (Implementation 2) 


-0.01 


0.02 


- 0 . 07 *** 


0.004 


0.03 


Year 2004-05 (Implementation 3) 




-0.03 


0.02 


Year 2005-06 (Implementation 4) 




- 0 . 04 ** 


- 0 . 09 *** 



Note 1 : These coefficients indicate the effect of the HPSGP on student achievement in standard deviations. Shaded 
cells Indicate that tests were not administered In all grades those years. 

* indicates statistical significance at 10%, ** at 5%, and *** at 1%. 



Exhibit 3.19 shows the HLM regression results for English learners. Even though the vast 
majority of students in these schools are of Hispanic origin, less than half are classified as 
English learners. Therefore, this is a relatively smaller group of students for analysis purposes. 



Exhibit 3.19. HLM Regression Results for English Learners in HP Only + Planning and 
Comparison Schools in a Large Urban District 



Year 


NRT 

Reading 


NRT Math 


NRT 

Language 


CST ELA 


CST Math 


Year 2000-01 


- 0 . 11 *** 


- 0 . 05 * 


- 0 . 12 *** 




Year 2001-02 (Planning) 


-0.01 


0.05* 


0.07** 




Year 2002-03 (Implementation 1) 


0.03 


-0.05 


0.01 


-0.004 


- 0 . 06 * 


Year 2003-04 (Implementation 2) 


0.02 


0.01 


- 0 . 12 *** 


-0.02 


0.03 


Year 2004-05 (Implementation 3) 




-0.03 


-0.02 


Year 2005-06 (Implementation 4) 




-0.02 


- 0 . 08 * 



Note 1 : These coefficients indicate the effect of the HPSGP on student achievement in standard deviations. Shaded 
cells Indicate that tests were not administered In all grades those years. 

* indicates statistical significance at 10%, ** at 5%, and *** at 1%. 



Contrary to the prior district results, we do not observe a positive effect on the NRT reading 
during the first year of the program. In line with the other analyses, however, CST math test 
scores show a statistically significant but small negative HPSGP effect during that year. 
Eurthermore, English learners in HP Only plus planning schools performed lower in relation to 
their counterparts in non-HP schools in NRT language in 2003-04 and in CST math in 2005-06. 
In the case of NRT language, the negative impact is more than twice that observed for all 
students in HP Only plus planning schools, still below the “small effect” threshold of 0.25 
standard deviations (Cohen, 1969). 

Einally, Exhibit 3.20 presents the HLM results for African-American students in this district. 
These students represent about 10 percent of the student population of interest. Given the much 
smaller number of observations, it is also more unlikely to observe statistically significant 
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48 

differences. Only the NRT math scores show differences between and HPSGP comparison 
groups. 



Exhibit 3.20. HLM Regression Results for African-American Students in HP Only + 
Planning and Comparison Schools in a Large Urban District 



Year 


NRT 

Reading 


NRT Math 


NRT 

Language 


CST ELA 


CST Math 


Year 2000-01 


-0.07 


0.06 


-0.13 




Year 2001-02 (Planning) 


-0.11 


-0.05 


0.04 




Year 2002-03 (Implementation 1) 


0.11 


- 0 . 15 ** 


-0.02 


-0.09 


0.03 


Year 2003-04 (Implementation 2) 


-0.06 


0.13* 


0.07 


-0.01 


-0.004 


Year 2004-05 (Implementation 3) 




-0.09 


-0.14 


Year 2005-06 (Implementation 4) 




0.05 


0.05 



Note 1 : These coefficients indicate the effect of the HPSGP on student achievement in standard deviations. Shaded 
cells indicate that tests were not administered in all grades those years. 

* indicates statistical significance at 10%, ** at 5%, and *** at 1%. 



Summary Findings Using Data from a Large Urban District 

Exhibit 3.21 presents the cumulative effect of the HPSGP using student-level longitudinally 
linked data from this district (i.e., summing all statistically significant differences of the overall 
results of Exhibit 3.16), alongside the statewide results. As observed, the comparison students 
outperformed students in HP Only plus planning schools in the district by 0.12 and 0.13 standard 
deviations in CSX EEA and math, respectively, since the program was implemented. Given that 
the standard deviation of these tests was in 2006 equal to 63.2 and 81.2 scale scores in EEA and 
math, respectively, students in comparison schools outperformed their counterparts in HP Only 
plus planning schools by about 7.6 points in EEA and 10.6 points in math. 

Exhibit 3.21. Cumulative Impact of the HPSGP in HP Only + Planning Schools, Statewide 
and for a Large Urban District 





CST ELA 


CST Math 


Statewide Elementary HP Only -i- Planning 


-0.02 


0.01 


Statewide Middle HP Only + Planning 


0.03 


0.02 


Statewide High HP Only -i- Planning 


-0.01 


-0.01 


Large Urban District HP Only -i- Planning 


-0.12 


-0.13 



Note: These figures are estimated summing all the statistically significant coefficients of Exhibits 3.10, 3.11, 3.12, and 
3.16. 

However, consistent with the statewide analyses, it is very important to note that these effect 
sizes - whether positive or negative - are too small to be considered “educationally significant” 
in either direction. In summary, the district-level results support the statewide analyses, in that 
the differences in student achievement between HP Only plus planning and comparison schools 
are negligible. 



Whenever sample size decreases, the power to reject null hypotheses, and therefore find statistically significant 
differences, also decreases. 
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Summary 

Across a wide array of alternative analyses, there is no educationally significant 
effect of participating in the HPSGP on student achievement. 

During the period of program implementation, HP Only plus planning schools, on average, 
showed gains in student performance, with more than twice as many students scoring proficient 
and above on the CST in 2006 than in 2001. At the same time, the best comparison group that we 
could construct for the purpose of this evaluation also exhibited similar growth. As discussed, the 
method used by the state in selecting the very lowest performing schools statewide for 
participation in the HPSGP precluded selection of an ideal comparison set of schools. (This is 
not to suggest criticism of this approach, but simply points out the analysis limitations it 
introduces.) This selection method unavoidably introduces some bias into the analysis. Given 
this, it is important to at least consider the possible effect of this known bias between the HP 
Only plus planning and comparison schools, even though it is not possible to fully determine it. 

In terms of other descriptive measures, a greater percentage of HP Only plus planning schools 
met their schoolwide API targets in two of the four implementation years in relation to the 
comparison schools. However, this trend was also present prior to program implementation. 
Contrary to this, a greater percentage of comparison schools met AYP in relation to HP Only 
plus planning. As these analyses do not control for student- and school-level characteristics, they 
can only be considered descriptive and not an assessment of the program’s impact. 

When controlling for these characteristics, the student-level achievement results vary across 
years and grade levels. Overall, the results suggest that HP Only and comparison schools 
performed virtually the same over the period of the program. These very small differences are 
supported also by analyses of student subgroups and longitudinally student-linked data from a 
large urban school district. 

Considering the empirical evidence presented in this chapter, questions arise as to why we 
observe no discernable impact on student outcomes given the fairly substantial amount of 
additional funding that the HPSGP provides (i.e., $400 per student per implementation year). 

The lack of impact seems fairly striking, even though prior literature has traditionally been quite 
mixed in terms of finding a strong link between educational inputs and outputs. This evaluation 
provides an opportunity to better understand some of the possible reasons for this weak link. 

These achievement analyses are based on a simple dichotomy - schools that did and did not 
participate in the HPSGP. However, within the group that participated in the program, our 
qualitative data collection shows considerable variation in implementation in key components, 
such as the external provider, and the extent to which funds were fully expended each year. The 
achievement analyses do not account for these differences - HP Only plus planning schools that 
may have implemented the program with fidelity are treated the same as schools that may have 
experienced breakdowns in implementation. In subsequent chapters, we turn to the findings from 

One recent example is the study “Successful California Schools in the Context of Educational Adequacy” carried 
out by Perez et al. ( http://irepp.stanford.edu/documents/GDF/STUDIES/17-AIR-Successful-Schools/17-Successful- 
Califomia-Schools(3-07).pdf ). The authors find no statistical evidence that beating-the-odds schools are radically 
different in terms of inputs than other schools in the state. 
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our qualitative research techniques to enrich our understanding of state, district and school 
practices that are likely to influence the degree to which HPSGP had an impact on student 
achievement. 



Page 66 



American Institutes for Research 




Evaluation of the High Priority Schools Grant Program 



Chapter 4: School and District Context for 
HPSGP Reform 



Introduction 

Before evaluating speeifie eomponents of the HPSGP and the pereeption of its impaet, this 
ehapter assesses the broader environment in whieh students learn and in whieh the program was 
implemented. As HPSGP does not exist in isolation, understanding the interaction between 
contextual factors (such as school capacity and the role of the district) with school improvement 
may help us identify conditions that lead to its successful implementation as well as reasons for 
its negligible overall impact. 

A primary goal of the Year 1 
site visits to 16 HP Only 
schools was to gain an 
understanding of the district 
and school factors that both 
hindered and facilitated the 
implementation of the 
HPSGP and the realization of 
its desired outcomes. The 
themes that emerged from the 
case studies were explored 
further across a broader 
sample of 106 schools and 49 
districts in the Year 2 survey 
data collection. Please see 
Chapter 2 for a review of 
these methods. 

In addition to building on 
case study findings, this 
chapter draws heavily from 
the results of school and 
district surveys, as well as 

supplemental data sources, such as statewide personnel resource data from CBEDS and the 
HPSGP Annual Reports. All school-level analyses for HPSGP schools pertain only to schools 



Key Findings 

• Staff levels do not appear to differ appreciably 
between HP Only and other schools statewide. 

• Survey evidence suggests that principal turnover 
negatively impacts the perception of school reform 
coherence. Respondents In schools with two or more 
principals over the last five years were more than twice 
as likely to believe that the school improvement program 
lacked coherence in comparison to respondents in 
schools with one principal. 

• Seventy percent of school respondents believed that 
the district was reasonably or very supportive. 
However, the majority of respondents did not believe 
their schools received differential district support in 
terms of administrative, instructional and support 
staff and non-personnel resources. 

• While a majority of surveyed school and district 
respondents reported that their district fostered a 
stable school administration, a considerable 
percentage (about 25 percent) disagreed. 
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that did not also participate in the II/USP or CSR - what we call in this report HP Only 
schools. 

We begin by assessing survey findings on school-level factors supporting school reform efforts, 
paying particular attention to personnel characteristics, such as quantity, attrition, and quality. 
We then explore the perceptions of district support - both current levels as well as support 
provided over the course of the grant period. The chapter concludes with a brief look at the 
district perspectives of their local school boards. Please note that the survey results reflect 
individuals’ perceptions (see Chapter 2 for types of respondents), which may not accurately 
represent the views of the school or district overall. 



School Contextual Factors 

This section reviews key school-level factors such as professional development and personnel 
quality that can directly shape schools’ improvement efforts. When interpreting these findings, it 
is important to keep in mind that HP Only schools, on average, serve greater proportions of 
students with special needs. As depicted in Exhibit 3.4 in Chapter 3, more than 80 percent of 
students in HP Only plus planning schools were of low socio-economic status, in relation to less 
the 50 percent of students statewide. HP Only plus planning schools also had more than twice as 
many English learners and students whose parents did not have a high school degree. These 
student demographics clearly pose an important, overarching factor that influence schools’ 
improvement efforts. 

Another predominant characteristic that sets these schools apart is their Program Improvement 
(PI) status under the NCEB. As described in Chapter I, failure to meet the federal AYP measures 
for two consecutive years results in schools being placed in PI, which features a broad range of 
sanctions and interventions that increase in severity each year that a school does not make AYP. 
Nearly 80 percent of HP Only schools are in Program Improvement, with nearly half of the PI 
schools in Year PI 4 or 5. By contrast, only 37 percent of all schools statewide are in PI, and of 
those, 30 percent are in Year 4 or 5. 



The Year 1 report for this evaluation referred to these schools as “Pure” schools. To make the label more intuitive, 
we call them “HP Only ” in this report. 

The responses for charter schools (n = 6) were set to missing for district support questions, due to their unique 
governance structure. This decision was supported by the fact that one charter left the district questions blank, 
explaining they were administered differently, and another charter noted in the survey narrative that its district 
responses pertained exclusively to their charter school which acts as its own district. 

Although Chapter 3 examined student demographics for HP Only plus planning schools, these patterns hold for all 
HP Only schools. 
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General School-Level Capacity 

While the survey respondents generally had positive perceptions of key school 
capacity characteristics, a considerable percentage indicated concerns with 
curriculum coordination and planning time. 

Research on instructional and organizational capacity and on professional development has 
examined the effects of school characteristics and capacity on student learning (Purkey & Smith, 
1983; Levine & Lezotte, 1990; O’Day, Goertz, & Floden, 1995; Newmann & Wehlage, 1995; 
Mohrman & Lawler, 1996; Williams, Kirst, & Haertel, 2005). These characteristics include a 
focus on student learning and common strategies, a culture of professional collaboration and 
collective responsibility, high quality curriculum, systematic monitoring of student learning, 
strong instructional leadership, and adequate resources. The Year 1 case study results identified 
several of these school-level factors as contributing to school improvement efforts. Some of the 
more predominant factors reported to facilitate academic success across the case study sites 
include the adoption and use of a common curriculum; a stable, strong, and collaborative 
teaching staff and administration; professional development opportunities; use of data to drive 
instruction and intervention programs; and the increased focus brought about through the 
scrutiny of accountability standards. 

While not intending to be a comprehensive review of all school-level factors, the school survey 
in Year 2 of this study built on these initial findings and posed a series of questions relating to 
vertical and horizontal instructional coherence through common curricula; staff motivation, 
collaboration, and professional development; staff content knowledge; coherence and 
consistency of school reform with clear goals. Exhibit 4.1 presents the results for these survey 
items. Although one might expect to observe differences by higher and lower performers (see 
Chapter 2 for these categories), no consistent statistically significant patterns emerged. 
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Exhibit 4.1 . School Respondents’ Perceptions on School Level Contextual Factors 



Source: HPSGP School and District Surveys (AIR) 



Over the past five years... 


Strongly 

Disagree 


Disagree 


Agree 


Strongly 

Agree 


No Answer 


a) There has been consistency in curriculum and instruction among teachers in the same grade level/subject area 


at this school. 












School Respondents, Overall (n = 106) 


1.9 


20.8 


55.7 


21.7 




Elementary (n = 44) 


0.0 


11.4 


61.4 


27.3 




Middle (n = 34) 


5.9 


29.4 


50.0 


14.7 




High (n = 28) 


0.0 


25.0 


53.6 


21.4 




b) Curriculum and instruction have been well coordinated across grade levels/subject areas. 


School Respondents, Overall (n = 106) 


1.9 


23.6 


58.5 


15.1 


0.9 


Elementary (n = 44) 


0.0 


22.7 


54.6 


20.5 


2.3 


Middle (n = 34) 


5.9 


23.5 


61.8 


8.8 


- 


High (n = 28) 


0.0 


25.0 


60.7 


14.3 


- 



c) staff at the school have been motivated and committed to making changes to improve student achievement. 



School Respondents, Overall (n = 106) 


0.9 


7.6 


54.7 


36.8 


- 


Elementary (n = 44) 


2.3 


6.8 


56.8 


34.1 


-- 


Middle (n = 34) 


0.0 


11.8 


44.1 


44.1 


- 


High (n = 28) 


0.0 


3.6 


64.3 


32.1 


- 



d) My school has had adequate teacher planning time to improve student achievement. 



School Respondents, Overall (n = 106) 


2.8 


28.3 


45.3 


23.6 


__ 


Elementary (n = 44) 


4.6 


18.2 


45.5 


31.8 


-- 


Middle (n = 34) 


0.0 


38.2 


38.2 


23.5 


-- 


High (n = 28) 


3.6 


32.1 


53.6 


10.7 


- 



e) My school has had adequate high-quality professional development to improve classroom instruction. 



School Respondents, Overall (n = 106) 


0.0 


17.0 


55.7 


27.4 


- 


Elementary (n = 44) 


0.0 


11.4 


56.8 


31.8 


- 


Middle (n = 34) 


0.0 


26.5 


41.2 


32.4 


„ 


High (n = 28) 


0.0 


14.3 


71.4 


14.3 


„ 



f) Most teachers in this school have had the knowledge and skills needed to teach the state standards. 



School Respondents, Overall (n = 106) 


0.0 


13.2 


59.4 


25.5 


1.9 


Elementary (n = 44) 


0.0 


9.1 


65.9 


20.5 


4.6 


Middle (n = 34) 


0.0 


23.5 


47.1 


29.4 




High (n = 28) 


0.0 


7.1 


64.3 


28.6 





g) We have had so many different initiatives in this school that our school improvement program lacks coherence. 



School Respondents, Overall (n = 


106) 


16.0 


56.6 


23.6 


3.8 




Elementary (n = 44) 




18.2 


59.1 


18.2 


4.6 


„ 


Middle (n = 34) 




20.6 


47.1 


29.4 


2.9 




High (n = 28) 




7.1 


64.3 


25 


3.6 




h) Student learning goals and strategies have been clearly defined in our school. 


School Respondents, Overall (n = 


106) 


0.0 


8.5 


66 


24.5 


0.9 


Elementary (n = 44) 




0.0 


6.8 


61.4 


31.8 


- 


Middle (n = 34) 




0.0 


17.7 


55.9 


23.5 


2.9 


High (n = 28) 




0.0 


0.0 


85.7 


14.3 


- 


i) The school’s strategies for improvement 


have been consistent (e.g., focus, use of strategies). 




School Respondents, Overall (n = 


106) 


1.9 


15.1 


54.7 


28.3 




Elementary (n = 44) 




0.0 


11.4 


54.6 


34.1 




Middle (n = 34) 




5.9 


23.5 


44.1 


26.5 




High (n = 28) 




0.0 


10.7 


67.9 


21.4 
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Coordinated curriculum 

Curriculum is featured in rows a) and b) of Exhibit 4.1. While more than three-quarters of the 
school-level respondents agreed that there had been consistency in curriculum and instruction 
within the same grade (row a), respondents from elementary schools were much more likely than 
middle sehool respondents to confirm this (88.6 percent to 64.7 percent). Lack of grade-level 
coherence appears espeeially problematic at the middle school level, where over a third of 
respondents indicated that their schools lacked within-grade consistency. Nearly three-quarters 
of all school respondents shared the perception of curriculum and instruction coordination across 
grades and subjeet areas (row b). 

While these findings seem positive, it is worthy of note that about a quarter of the respondents 
indicated that such coordination is not occurring. This seems somewhat disconcerting for schools 
undergoing the kinds of intensive school reform efforts that HPSGP is designed to produce. 

Staff motivation, staff planning and professional development 

The vast majority (91.5 percent) of all HP Only school respondents believed that staff were 
motivated and committed to improving student learning (row c in Exhibit 4.1). Despite the noted 
importance of joint planning time at the case study schools, more than 30 percent of all surveyed 
sehool respondents did not believe that their schools had adequate planning time (row d). About 
84 percent overall believed that the school had adequate, high quality professional development 
(row e), with middle school respondents statistically significantly less likely than elementary 
school respondents to agree with item e (74 percent to 89 percent). In terms of content 
knowledge (row f), middle school respondents again were statistically significantly less likely 
than high sehool respondents to believe that most teachers had the knowledge and skills need to 
teach to the state standards (77 percent to 93 pereent).^^ 

Consistency and coherence of school reform initiatives 

Given that some of the case study schools experienced chronic disruptions in their school reform 
efforts, often attributed to changes in school leadership, the survey assessed the extent to which 
HP Only school respondents believed that the school reform program was coherent and 
consistent, and had clear goals. While a majority of surveyed schools disagreed, a quarter of the 
school respondents believed that the sehool had so many initiatives that the improvement 
program lacked coherence (row g). 

However, this perceived lack of coherence did not seem to influence the respondents’ views on 
whether student learning goals and strategies were clearly defined. Overall, more than percent 
believed the goals and strategies were clear, although middle sehool respondents were less likely 
than those from high sehools to agree with this (79.5 percent versus 100 percent). Respondents 
from both high schools and elementary schools were more likely than middle sehool respondents 

C"7 

to believe that that the school’s improvement strategies are consistent. The relationship 



Significant at the 1 percent confidence level. 

Significant at the 10 percent confidence level. 

Significant at the 10 percent confidence level. 

Significant at the 5 percent confidence level. 

Significant at the 5 percent confidence level for elementary/middle school comparisons, and 10 percent for 
high/middle school comparisons. 
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between prineipal turnover and sehool reform eoherenee, as measured by these items, will be 
discussed in the subsequent section on personnel. 

In summary, while the survey results generally pointed to positive perceptions of key school 
capacity characteristics, a sizeable percentage of the school respondents expressed concerns with 
coordination of the curriculum, and sufficient planning time. While most respondents seemed to 
agree that the professional development was adequate, the overall knowledge and skill level of 
their staff was sufficient to teach to state standards, and the reform strategies were consistent, 
middle school respondents were less confident with about one-quarter indicating that this was 
not the case. 



Personnel Resource Capacity 

While professional development, planning time, and instructional coherence are important 
strategies, the potential of these strategies for building capacity and raising student achievement 
will be directly impacted by personnel resources, such as staffing levels, quality, and turnover. 

To inform our understanding of these factors, we drew upon statewide CBEDS data, HPSGP 
Annual Reports, and survey results. 

Staffing Levels 

Staff levels do not appear to differ appreciably between HP Only and other schools 
statewide. 

An important preface to this section is that increasing staffing levels was not a specified program 
goal of the HPSGP. At the same time, as will be shown in Exhibit 5.13 in Chapter 5, nearly half 
of the school survey respondents reported “supplemental personnel” (e.g., teachers, literacy 
coaches) as the largest and as the most effective use of HPSGP funds at their schools. Given this, 
it might be reasonable to expect staffing levels at HPSGP schools to exceed what is found on 
average across all schools statewide. 

In addition, as shown in Chapter 3, HP Only plus planning schools face some of the most 
academically challenged student populations in the state, with considerably higher percentages of 
students in poverty and English learners. As outlined in Berne and Stiefel (1994) and as 
determined in virtually all education adequacy determinations conducted across the states 
(www.cfequity.org), sites with higher percentages of students with supplemental learning needs 
(e.g., poverty and EE) have generally been determined to require more resources than schools 

CO 

with fewer numbers of these students when expected to reach common educational outcomes. 
That is, equal funding generally has not been considered “equitable” across sites with 
measurably different student needs. Consequently, in order to be considered as being treated 
equally given the high needs student populations HPSGP schools serve, greater personnel 
resources might be expected at these schools prior and in addition to any supplement received 
through the HPSGP grant. 



This was also found in all three of the separate adequacy studies conducted for California this past year as a part 
of the Getting Down to Facts studies described in greater detail in Chapter 7 of this report. 
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From a statewide perspeetive, however, HP Only sehools do not appear to differ eonsiderably in 
terms of overall full-time equivalent (FTE) personnel levels in relation to the eomparison sehools 
used for the achievement analyses and all other schools in the state. As shown in Exhibit 4.2, 
HP Only schools, on average, had only slightly more overall, teacher, and administrator PTEs in 
2005-06. As schools receive support from district-level staff, such as subject matter coaches, 
which cannot be tracked to specific schools in CBEDS, these analyses may underestimate the 
overall personnel available to the schools. Note, however, this would apply to both HP Only and 
non-HP schools, to the extent that they receive support from the district or other personnel not 
directly associated with the schools in CBEDS. 

While we observe an increase in overall PTE levels among HP Only schools from 2001-02 to 
2005-06, teacher PTEs declined across all school groups during this period. Even with this 
decline, however, HP Only schools had a total of 4.68 PTE teachers per 100 students, above the 
average of 4.41 for comparison schools and 4.64 for all other schools. Por a school with 800 
students - the average school size of HP Only schools in 2005 - this translates to a difference of 
approximately 2.2 more PTE teachers in HP Only schools, in relation to comparison schools (and 
an additional 0.7 PTE in relation to all other schools). 



Exhibit 4.2. Average FTE Personnei per 100 Students by School Grouping, 2001-02 
and 2005-06 

Source: CBEDS PAIF (Personnel Assignment Information File) and SIF (Student Information File), 2001-02 through 2005-06. 



2001-02 


2005-06 






Total PTEs Per 100 




Personnel Type 


N of Schools 


Students 


Total PTEs Per 100 Students 


Total FTE (all staff) 


HP Only 


341 


5.16 


5.24 


Comparison 


90 


5.03 


4.97 


All Other 


10,022 


5.19 


5.15 


Teachers 


HP Only 


341 


4.70 


4.68 


Comparison 


90 


4.52 


4.41 


All Other 


10,022 


4.74 


4.64 


Administrators 


HP Only 


341 


0.24 


0.35 


Comparison 


90 


0.26 


0.31 


All Other 


10,022 


0.25 


0.30 


Pupil Support Staff 


HP Only 


341 


0.22 


0.21 


Comparison 


90 


0.25 


0.25 


All Other 


10,022 


0.20 


0.22 



The Year 1 report found that HP Only schools had, on average, lower levels of personnel. However, the results 
were based on a different comparison group and used unweighted averages which were more susceptible to 
enrollment variations. These personnel categories are aligned with administrative employees, teachers and pupil 
services employees as defined in Education Code 41401 and referenced in the Administrative Manual for CBEDS 
Coordinators and School Principals, October 2005. Please refer to Technical Appendix D for a more detailed 
description of the specific personnel assignments that are included within each category. 

The average school size of HP Only schools was 796 students, using the 2005 API Growth database. 
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Note 1 : These figures exclude alternative, continuation, special education, state special, juvenile hall, community day or adult 
education schools. 

Note 2: In deriving the FTE per 100 students, we summed all FTEs (by type) for each school grouping and divided by the total 
enrollment in that group. 



While these results suggest that HP Only schools may have a slight advantage in teacher levels, 
they do not take into account differing student demographics. Showing the number of students 
per teacher and pupil support staff, Exhibit 4.3 presents striking differences between HP Only 
and other schools. While the number of overall students per teacher is fairly similar between the 
school groups, HP Only schools have considerably higher numbers of low socio-economic status 
students and English learners per teacher in relation to all other schools. Specifically, teachers in 
HP Only schools serve 2.5 more low socio-economic status students than teachers in comparison 
schools, and nearly 8 more than teachers in all other schools. A similar pattern holds true for 
English learners, with teachers in HP Only schools serving twice as many English learners than 
teachers in all other schools. The differences for pupil support staff are even more pronounced, 
with HP Only schools serving in excess of 125 more students with low-socioeconomic status in 
relation to comparison and all other schools. Any personnel advantage observed above in Exhibit 
4.2 is likely to be offset by differences in the number of students with special learning needs 
served. 

Exhibit 4.3. Average Number of Students per FTE Teacher and Pupil Support Staff, 
2005-06 

Source: Figures based on CBEDS PAIF (Personnel Assignment Information File) and SIF (Student Information File), 2005-06. 
English learner enrollment estimates derived from the API Growth Database. 



Number of Students per FTE Staff 
2005-06 Averages 


HP Only 
Schools 

(N = 338)* 


Comparison 

Schools 

(N = 90) 


All Other 
Schools 

(N = 9,869)* 


Overall Students per Teacher 


21.4 


22.7 


21.5 


Free/Reduced Price Lunch Students per Teacher 


17.7 


15.5 


10.1 


English Learners per Teacher 


9.8 


8.1 


4.7 


Overall Students per Pupil Support Staff 


480.4 


396.7 


462.0 


Free/Reduced Price Lunch Students per Pupil 
Support Staff 


397.3 


270.9 


216.0 


English Learners per Pupil Support Staff 


220.4 


141.5 


100.0 



* The numbers of observations are lower than Exhibit 4.2 due to missing data. 



In summary, while overall and teacher staffing levels did not appear to differ appreciably 
between HP Only and other schools statewide, it raises questions as to how these schools can be 
expected to make academic progress with similar levels of staff resources as schools serving less 
challenging populations. One might suggest that this is compensated through higher teacher 
quality, which is very difficult to measure. As one approximation, however, additional findings 
presented later show that HP Only schools also have lower proportions of fully credentialed 
teachers. 
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Staff Turnover 

Survey evidence suggests that principal turnover negatively impacts the perception 
of school reform coherence. Respondents in schools with two or more principals 
over the last five years were more than twice as likely to believe that the school 
improvement program lacked coherence in comparison to respondents in schools 
with one principal. 

Staff turnover - both in terms of leadership and instructional staff - was frequently cited by the 
case study schools as problematic to school reform efforts. Using information from the survey 
and the HPSGP Annual Reports, this section examines district and school staff attrition, as well 
as the perceived impact of attrition on school improvement efforts. However, since statewide 
turnover data are not available from CBEDS, we are unable to make comparisons between HP 
Only schools with other groups of schools. 

School Leadership Turnover 

A 2003 RAND study examining the 1999-2000 School and Staffing Survey (SASS) found that 
although national and state data for years of principal experience on average suggest relative 
stability for school administrators, certain types of schools and districts encounter challenges in 
recruiting and retaining principals. For example, some urban, low-income schools were found to 
have trouble keeping experienced principals, although this pattern was not seen across all urban, 
low-income schools (Gates, Ringel, Santibanez, Chung, & Ross, 2003). Consistent with these 
results, a study by Roza, Celio, Harvey, and Wishon (2003) which surveyed 83 urban school 
districts (including 29 California districts) found that districts and schools with the fewest 
applicants for positions are typically “high need” — with lower median community income levels, 
higher concentrations of minority students, and lower principal salaries as compared to less high 
need schools and districts. 

Using information in the HPSGP Annual Reports, Exhibit 4.4 displays the percentage of HP 
Only principals by number of years that they have been at the current school from 2002-03 to 
2004-05. These data suggest that a substantial percentage of HP Only schools have relatively 
new principals. Across all years, nearly 30 percent of principals were at their current HP Only 
school for less than a year, indicating a high degree of turnover. The percentage of principals 
with three years or less at their current school hovered around 60 percent. A more encouraging 
sign is the increase in the percentage of principals at their school for more than five years, from 
22 to 27 percent. Among the 14 principals at the surveyed schools that were new to the school 
(e.g., one year or less), 21 percent were first-time principals - similar to the proportion of first 
year principals observed in the case study schools. 
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Exhibit 4.4. Percentage of Principals by Years at Current HP Only School, 2002-03 
through 2004-05 

Source: HPSGP Annual Reports, 2002-03 through 2004-05. 

N = 292 N = 290 N = 297 








2003 - 04 




> 5 Years 
V' 4 - 5 Years 
□ 3 - 4 Years 

■ 2-3 Years 

■ 1 - 2 Years 

■ 0-1 Years 

2004 - 05 



15% 

12 % 



30% 



Year 



The finding that a considerable percentage of HP Only schools appeared to have more than one 
principal over the first three years of the program was consistent with the level of turnover 
observed in the case study schools. While 8 of the 16 case study schools had reasonably well- 
established and continuous leadership, half of the schools (in six districts) did not. In four cases, 
changing leadership was a chronic problem This included schools with four principals in the last 
three years; ten in the last twelve years; five in the last three years; and six principals in the last 
five years. Three other sites had new, first-year principals, and one site had not had a principal at 
the school thus far in the school year and did not expect one to be appointed until the next school 
year. While new HPSGP principals (i.e., a year or less at the site) reported an average of eight 
years experience as a principal or administrator elsewhere (as reported by the 2004-05 HPSGP 
Annual Report) the degree of new leadership at the case study schools seemed exceptionally 
high. While five of the eight sites with unstable leadership were actually making their annual 
API targets, the lack of leadership was cited by a number of respondents as a major disadvantage 
in relation to what might have been accomplished. 

The HP Only schools that were surveyed reported turnover rates that ranged from 1 to 4 
principals over the five years of the implementation period of the grant. More than a third of the 
schools had a single principal over five years; another 38 percent had two principals; 21 percent 
reported three principals; and the remaining 4 percent had four principals during that time period. 
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However, these changes in school leadership were not consistently reported as an obstacle for 
school improvement efforts. For example, as shown in Exhibit 4.5, half of the schools with two 
principals and more than quarter of schools with three or more principals described this degree of 
change as facilitating improvement efforts. However, these data may be difficult to interpret, as 
respondents may have been considering the skills of the last principal as opposed to the impact of 
multiple leadership changes. 

On the other hand, a considerable percentage of respondents (43 percent, or 28 of the 66 schools 
with two or more principals) reported that the change in leadership was somewhat disruptive or a 
major impediment. Additionally, in response to a separate narrative question regarding factors 
that limited the school’s ability to use the HPSGP funds, six school respondents identified 
principal turnover as a factor. As one school respondent described, “Turnover in leadership 
disrupted the coherence of the reforms as each new principal had to get up to speed on the plan, 
the tenets of the HPSGP in addition to becoming familiar with his/her new school and staff.” 

We acknowledge that other leadership positions may have experienced significant attrition and 
although these changes were not captured by the survey they were still important. As described 
by one school respondent, “We have been through 3 principals and 8 assistant principals, due to 
movement in the district, not by choice or promotion.” 
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Exhibit 4.5. School Respondents’ Perception of the Impact of Change in Principals 
Over the Past Five Years on School Improvement Efforts 

Source: HPSGP School Surveys (AIR) 



Impact of change on improvement efforts... 


Number of principals over the past five years... 
1 2 3 + 




(n = 39) 


(n = 40) 


(n = 26) 


Facilitated our school improvement efforts 


67% 


50% 


27% 


No Impact on school Improvement efforts 


28% 


28% 


0% 


Somewhat disruptive 


0% 


18% 


42% 


A major impediment to our school Improvement efforts 


3% 


5% 


31% 



<1> The responses for schools with 3 and 4 principals over the past five years were grouped together due to small number of 
observations. 

Note: Percentages will not add to 100 percent due to rounding and missing responses. 



Linking the level of principal turnover to perceptions of reform consistency and clear goals (see 
Exhibit 4.1), we found additional evidence that changes in school leadership has negative 
implications for improvement efforts. Respondents in schools with two or more principals over 
the last five years were more than twice as likely to believe that the school had so many 
initiatives that the improvement program lacked coherence (35 percent), in comparison to 
schools that had one principal (15 percent). In addition, a statistically significantly greater 
percentage of respondents in schools with one principal (97 percent) believed that student 
learning goals and strategies were clearly defined, in relation to other schools (88 percent). 
Although a greater percentage of respondents from schools with one principal believed that the 
school’s improvement strategies have been consistent - 90 percent in relation to 79 percent of 
schools with two or more principals over time - this difference was not statistically significant. 

Instructional Staff Turnover 

Teacher turnover was generally cited as a concern by principals and teachers across the majority 
of the case study schools. In the most recent year, nearly a third of the surveyed schools reported 
minimal turnover (e.g., less than 10 percent of current staff is new). However, nearly as many 
reported between 10 and 19 percent of their teaching staff as new, and between 20 and 39 
percent of the staff were new in a quarter of the surveyed schools. On the extreme end, five 
schools reported 40 percent or more new staff. 



Significant at the 5 percent confidence level. 
® Significant at the 10 percent confidence level. 



Page 78 



American Institutes for Research 





Evaluation of the High Priority Schools Grant Program 



Exhibit 4.6. Percentage of Teaching Staff New to the School in 2006-07, as 
Reported by Surveyed School Respondents 



Source: HPSGP School Surveys (AIR) 









% of schools that reported that the % of new staff in 
2006-07 was... 


% New Teaching 
Staff 


N 


% of Surveyed 
Schools 


Lower than the 
last five years 


Typical 


Higher than the 
last five years 


0-9 


37 


35% 


57% 


41% 


3% 


10-19 


36 


34% 


33% 


61% 


6% 


20-29 


15 


14% 


40% 


47% 


13% 


30-39 


12 


11% 


17% 


42% 


42% 


40-69 


5 


5% 


0% 


60% 


40% 


Missing 


1 


1% 


-- 


-- 


-- 


Total 


106 


100% 









The majority of schools that reported turnover between 0 and 9 percent indicated that this was 
lower than what the school experienced over the past five years, while the majority of schools 
with 10 to 19 percent new staff reported this level of turnover was typical. Of concern are three 
schools with turnover in the range of 40 to 69 percent, for which this appears to be a chronic 
issue. 



School respondents were asked to characterize the impact of teacher turnover between 2002-03 
and 2006-07 (i.e., the HPSGP grant period) on their school improvement efforts. Given that 
teacher attrition varied over this time. Exhibit 4.7 presents these results for schools that indicated 
the percentage of new teaching staff in 2006-07 was representative of the last five years. As the 
percentage of new teachers increases, so does the percentage of schools reporting turnover to be 
a major impediment. However, a third of respondents in schools with 10 to 19 percent new staff 
described this change as facilitating school improvement efforts, suggesting that there may be an 
optimal level of turnover that provides a balance between new, and likely inexperienced, staff 
and veteran staff who may be less inclined to embrace reforms. 



Exhibit 4.7. School Respondents’ Perception of the Impact of Change in Teachers 
Over the Past Five Years on School Improvement Efforts (in schools for which the 
2006-07 teacher turnover was “typical” of the last five years) 



Source: HPSGP School Surveys (AIR) 



Impact of change on improvement efforts over last five 
years... 



% of new teachers in most recent year... 
0-9% 10-19% 20-69% 

(n = 15) (n = 22) (n = 15)'^’ 



Facilitated our school improvement efforts 


20% 


36% 


13% 


No impact on school improvement efforts 


73% 


23% 


13% 


Somewhat disruptive 


7% 


32% 


40% 


A major impediment to our school improvement efforts 


0% 


5% 


25% 



<1> The responses for schools reporting between 20 and 69 percent teacher turnover were grouped together due to small number 
of observations. 

Note: Percentages will not add to 100 percent due to rounding and missing responses. 
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District Superintendent Turnover 

In addition to site level staff, turnover in distriet leadership may influenee the pereeption of 
distriet support and eoherenee between sehool and distriet efforts. Prineipal and teaeher 
respondents in four of the nine distriets visited as a part of the case studies mentioned that 
administrative turnover at the district level posed a major challenge in their efforts to improve 
student performance. The survey results showed that a third of the districts in our school sample 
had one superintendent over a five year period, while nearly half had two superintendents, and 
another 17 percent had three or more.^^ As expected, fewer school respondents perceived the 
change as facilitating improvements as the number of new superintendents increased (see Exhibit 
4.8). The most striking difference is the jump in the percentage of schools that view the change 
as a major impediment - 5 percent of schools with 2 superintendents and 62 percent of schools 
with 3 or more. 



Exhibit 4.8. School Respondents’ Perception of the Impact of Change in District 
Superintendents 

Source: HPSGP School and District Surveys (AIR) 



Impact of change on improvement efforts... 


Number of superintendents over the past five years... 
1 2 3 + 

(n = 20) (n = 71) (n = 13)'^" 


Facilitated our school Improvement efforts 


85% 


33% 


8% 


No Impact on school Improvement efforts 


15% 


38% 


8% 


Somewhat disruptive 


0% 


24% 


23% 


A major impediment to our school 
improvement efforts 


0% 


5% 


62% 



<1> The responses for schools that reported with 3-5 district superintendents over the past five years were grouped together due to 
small number of observations. 

Note: Percentages will not add to 100 percent due to rounding. 



Although we cannot compare these patterns of turnover to those in other schools in the state, as 
CBEDS does not track this information, the school survey results nonetheless indicate that 
district- and school-level staff attrition appears to have an impact on improvement efforts in HP 
Only schools. 

Staff Quality 

While HP Only schools are slightly below other schools statewide in terms of 
teacher credentials and education levels, the vast majority of surveyed district 
respondents believed that teacher quality in HPSGP schools was as good as the 
district average. 

The issue of personnel stability is also closely associated with the quality of existing and new 
staff in terms of credentials, education, and experience. Again turning to CBEDS for a statewide 
picture, the percentage of ETE teachers with full credentials increased considerably across all 
school groups from 2001-02 to 2005-06 (Exhibit 4.9). With the lowest percentage of credentialed 



63 



Based on 106 school surveys, representing 46 districts. One district had five superintendents in five years. 
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teachers in 2001-02, HP Only schools made the greatest gains over time, surpassing the 
comparison schools by a slim margin in 2005-06, but remaining below the average of all other 
schools. 

In terms of education levels, we again observe increases in the percentage of teachers with a 
masters or doctorate for all school groups. While the gap between HP Only and the other groups 
declined over this period, HP Only schools continued to have lower percentages of teachers with 
this level of education by 2005-06. With respect to experience levels, HP Only schools showed 
marginally lower number of years of experience among teachers. 



Exhibit 4.9. Average Percentage of Teachers with Fuil Credentials, Education 
Level, and Years of Experience by School Grouping, 2001-02 and 2005-06 



Source: CBEDS PAIF (Personnel 


Assignment Information 


I File) and SIF (Student Information File), 


2001-02 through 2005-06 




N of schools 


2001-02 


2005-06 


FTE Teachers with Credentials % % 


HP Only 


341 


72.7% 


90.3% 


Comparison 


90 


78.7% 


89.0% 


All Other 


10,022 


87.3% 


94.2% 


FTE Teachers with Masters or 


Doctorate 


% 


% 


HP Only 


341 


25.4% 


29.0% 


Comparison 


90 


31 .2% 


32.1% 


All Other 


10,022 


30.4% 


32.9% 


FTE Years of Experience 




Mean 


Mean 


HP Only 


341 


10.9 


11.1 


Comparison 


90 


12.1 


11.8 


All Other 


10,022 


12.6 


12.5 



Note: These figures exclude alternative, continuation, special education, state special, juvenile hall, community day or adult 
education schools. 



Given that the HPSGP schools were generally the lowest performing schools statewide, we 
might expect that the quality of their staff might have received particular attention from the 
district during the grant period in relation to other schools in the district. While the personnel 
analysis based on CBEDS above provides a picture of how HP Only schools compare to other 
schools statewide, it does not account for intra-district variations. Recent research by Education 
Trust-West (2005) has shown substantial funding gaps in California between high and low 
poverty schools in the same district. The report shows these gaps, which disadvantage high 
poverty schools, result from concentrations of more experienced and more highly credentialed 
teachers in lower poverty schools. 

As HPSGP schools (as shown in Chapter 3, for HP Only plus planning schools) on average serve 
higher poverty and higher minority students in relation to the statewide average, these intra- 
district variations are of concern. In our survey, we asked district respondents to describe the 



Limited resources prevented us from analyzing personnel distributions within districts. While we raised intra- 
district analyses as a possible option, the CDE and Advisory Committee members decided to pursue longitudinally- 
linked student level achievement analyses instead. 

Furthermore, nearly half of the largest school districts that Education Trust - West identified as spending more on 
average teacher salaries in schools in the bottom poverty quartiles participated in our survey. 
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overall quality of school administrators and teachers, as well as specifically their experience, 
education, and content knowledge, in relation to the district average. As shown in Exhibit 4.10, 
the vast majority of the respondents believed the overall quality of these staff was as good as the 
district average. 

Exhibit 4.10. District Respondents’ Perceptions of the Quality of School 
Administrators and Teachers in HPSGP Schools, in Relation to the District 
Average (n = 49) 



Source: HPSGP District Surveys (AIR) 





As Good 
As 


Better Than 


Worse Than 


No Answer 


Overall School Administrator Quality 


75.5 


12.2 


0.0 


12.2 


In terms of experience 


65.3 


14.3 


12.2 


8.2 


In terms of education 


77.6 


14.3 


0.0 


8.2 


In terms of content knowledge 


73.5 


14.3 


4.1 


8.2 


Overall Teacher Quality 


87.8 


2.0 


2.0 


8.2 


In terms of experience 


71.4 


6.1 


12.2 


10.2 


In terms of education 


83.7 


2.0 


4.1 


10.2 


In terms of content knowledge 


75.5 


12.2 


4.1 


8.2 



To summarize this section on the school context, while HP Only schools were generally positive 
about some key capacity characteristics (e.g., teacher planning and professional development), 
we have concerns regarding their most critical resource - personnel. Although personnel levels at 
HP Only schools do not appear to substantially differ from other schools, the fact that these 
schools serve populations with greater needs (as depicted in Exhibit 3.4 in Chapter 3 and Exhibit 
4.3) suggests that the same allocations of staff as all schools may be insufficient to address the 
substantially greater challenges these schools face. To meet these challenges, districts may need 
to actively provide differential levels of support in HPSGP schools, a topic explored further in 
the following section. 



Role of the District in School Reform 

A growing body of research has put increasing emphasis on the role of the district in school-level 
reform efforts. Marsh (2000) found that districts can play a “potentially critical role in improving 
teaching and learning.” Districts acting as change agents tend to mobilize critical resources, 
increasing the likelihood “of enacting and sustaining state and local reform goals and policies.” 
The coherence of school programs and district goals has also been identified as an important 
quality of effective schools and districts. A recent study of high poverty elementary schools in 
California found that “performance is higher in schools in which the actions of teachers, 
principals, and school district officials are all closely aligned and tightly focused on student 
achievement” (EdSource, 2006). 
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General District Support 

Surveyed school respondents had generally positive views of the district, with 70 
percent believing the district was reasonably or very supportive. However, despite 
their participation in a program targeting the lowest performing schools in the state, 
the majority of respondents did not believe their schools received differential 
district support in terms of administrative, instructional and support staff and non- 
personnel resources. 

In Year 1, one of the predominant themes surfacing from the case studies was the critical and 
substantially varying role of the district. Varying degrees of district support clearly influenced 
the visited schools’ ability to address challenges in implementing the HPSGP and to improve 
student performance. Given that schools were selected into the program due to low performance 
and that a substantial financial commitment with important sanctions were associated with their 
participation, we expected a certain degree of special attention to and focus on these schools by 
their district offices. Often, however, this seemed not to be the situation in our case study 
schools, raising important questions about the ability of interventions like the HPSGP to make an 
impact on a school even on a short-term basis without clear supporting action from the district. 
Findings from the case studies raised additional concerns about the long-term sustainability of 
realized gains in the absence of substantial district buy-in and involvement. 

Among the nine districts visited during the case studies, based on extensive interviews with 
district respondents and at the visited schools, the study team assessed that a third were engaged 
and helpful in assisting schools to address their academic challenges, specifically in regard to 
HPSGP implementation. Four of the nine districts visited, however, were generally characterized 
as not helpful during the time of program implementation, three of which were or recently had 
been in some form of crisis. In these cases, the district office seemed to constitute one of the 
major challenges the schools faced in improving student achievement, which respondents at the 
district office as well as at the schools tended to acknowledge. Based on what we saw and heard 
during these visits, the remaining districts were assessed as neither a major help nor detriment to 
school improvement efforts. The perception of school respondents in these districts was that 
areas of assistance were fairly evenly offset by areas in which the district was seen to be holding 
them back. In relation to the level of support that might be expected for some of the state’s most 
academically challenged schools, overall perceptions of district support in the case studies 
suggested room for improvement. 

School and district surveys administered in Year 2 explored perspectives on current and prior 
levels of district support across a larger sample. District respondents were asked to select one of 
the three options provided in Exhibit 4.1 1 that best described the district’s goals and support of 
school improvement activities in low performing schools. The results were split between districts 
reporting that they had clear goals with a focus on supporting school reform (5 1 percent) and 
those that were in the process of developing, or revisiting, their goals and support (49 percent). 
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Exhibit 4.11. Current District Context as Perceived by District Respondents 

Source: HPSGP District Surveys (AIR) 



In your opinion, which of the following three situations best describes the 
current context in your district with regard to low performing schools? 





N 


Percent 


The district has clear, cohesive goals and is focused on 
supporting school Improvement. 


25 


51.0 


The district Is developing or reforming school improvement 
goals and support. 


24 


49.0 


The district does not have clear school Improvement goals and 
support. 


0 


0.0 



School responses to a similar question on current district support were generally more positive 
than what was observed at the case study sites, as described above. Seventy percent of all school 
respondents indicated that the district was reasonably or greatly supportive of their improvement 
efforts (Exhibit 4.12). At the same time, however, only about a quarter of the respondents 
described the district as greatly helping. Moreover, 13 percent reported that the district neither 
helped nor hurt; 6 percent said the district failed to provide any support; and 4 percent described 
their districts as seriously hurting their efforts. This suggests that districts could be doing more to 
actively assist their most academically challenged schools, where it might be expected that 
district support efforts would be the greatest. These views were relatively consistent across 
school types, while elementary school respondents were slightly more likely to hold a negative 
view of district support and middle schools were more likely to be neutral. 

Exhibit 4.12. Current Level of District Support as Perceived by School 
Respondents 

Source: HPSGP School Surveys (AIR) 

How would you describe the central district's current level of support on your school improvement efforts? 





Seriously 
hurts our 
efforts 


Fails to 
provide any 
support 


Neither 
helps nor 
hurts our 
efforts 


Provides a 
reasonable 
level of 
support 


Provides 
support that 
greatly helps 
our efforts 


No 

Answer 


School Respondents, 
Overall (n = 106) 


3.8 


5.7 


13.2 


42.5 


27.4 


7.6 


Elementary (n = 44) 


4.6 


6.8 


9.1 


40.9 


29.6 


9.1 


Middle (n = 34) 


2.9 


5.9 


17.7 


44.1 


23.5 


5.9 


High (n = 28) 


3.6 


3.6 


14.3 


42.9 


28.6 


7.1 



We further probed school respondents’ perceptions on whether district’s goals, policies, and 
procedures facilitated school reform. Approximately 80 percent of the surveyed school 
respondents believed that the district had set clear, consistent goals and provided critical 
expertise in supporting improvement efforts (Exhibit 4.13). Respondents from HP Only high 
schools were most likely to perceive the importance of the district staff’s expertise in relation to 
other school types, with nearly 93 of the respondents agreeing or strongly agreeing on this item. 
While about two-thirds overall did not believe the district hindered important decisions and the 
school’s ability to address student needs, about a quarter of the school respondents indicated that 
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the district interfered (with 15 percent of middle school respondents strongly agreeing that 
district policies and procedures made it difficult to address the needs of their students). 



Exhibit 4.13. Current District Goals, Expertise, and Policies as Perceived by 
School Respondents 



Source: HPSGP School Surveys (AIR) 



Strongly 

Disagree 



Disagree 



Agree 



Strongly No 
Agree Answer 



The district sets clear, consistent goals for schools. 



School Respondents, Overall (n = 106) 


1.9 


12.3 


47.2 


33.0 


5.7 


Elementary (n = 44) 


2.3 


11.4 


43.2 


36.4 


6.8 


Middle (n = 34) 


0.0 


11.8 


55.9 


26.5 


5.9 


High (n = 28) 


3.6 


14.3 


42.9 


35.7 


3.6 


District staff provide important information and expertise that supports our school improvement efforts. 


School Respondents, Overall (n = 106) 


1.9 


12.3 


50.0 


29.3 


6.6 


Elementary (n = 44) 


2.3 


13.6 


47.7 


29.6 


6.8 


Middle (n = 34) 


2.9 


17.7 


41.2 


29.4 


8.8 


High (n = 28) 


0.0 


3.6 


64.3 


28.6 


3.6 


The district's poiicies and procedures make it difficult for us to address the specific needs of students at 


our school. 












School Respondents, Overall (n = 106) 


22.6 


44.3 


18.9 


8.5 


5.7 


Elementary (n = 44) 


22.7 


43.2 


20.5 


6.8 


6.8 


Middle (n = 34) 


17.7 


41.2 


20.6 


14.7 


5.9 


High (n = 28) 


28.6 


50.0 


14.3 


3.6 


3.6 


The district interferes with our abiiity to make important decisions at this schooi. 


School Respondents, Overall (n = 106) 


21.7 


47.2 


16.0 


8.5 


6.6 


Elementary (n = 44) 


20.5 


47.7 


13.6 


9.1 


9.1 


Middle (n = 34) 


14.7 


52.9 


17.7 


8.8 


5.9 


High (n = 28) 


32.1 


39.3 


17.9 


7.1 


3.6 



In addition to the current district context, respondents were asked to consider district support 
over the duration of the grant period (i.e. between 2002-03 and 2006-07). About 62 percent of 
the school respondents believed that the district was a critical factor in improving student 
achievement over the past five years. While this seems a favorable reflection of district 
assistance, it is noteworthy that 30 percent of school respondents disagreed that the district had 
been a critical factor. 

As depicted in Exhibit 4.14, twice as many middle school respondents disagreed that the district 
was a critical factor, in relation to high school respondents. While the discrepancy between 
district and school respondents appears substantial, the reader is reminded that these groups are 
not entirely comparable (e.g., there are schools for which we did not obtain a district survey, and 
vice versa). However, when limiting the analysis to respondents for which we obtained both 
school and district surveys, 65 percent of the school respondents believed the district was a 
critical factor, in relation to 88 percent of the district respondents, suggesting a considerable 
disconnect between these perceptions. 
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Exhibit 4.14. School and district responses to “Over the past five years, the district was a 
critical factor in improving student achievement” 

Source: HPSGP School and District Surveys (AIR) 




Aside from district expertise and guidance, districts provide other important support in the form 
of school-level personnel and non-personnel resources. Shown in Exhibit 4.15, while nearly 90 
percent of the district respondents agreed that the district had provided adequate resources over 
this time period, this was true for only 64 percent of the school respondents. When limiting the 
analysis to schools for which we obtained both school and district surveys, the differences 
remain pronounced (89 percent versus 67 percent). Again, the largest school-level difference was 
between middle and high schools, where a quarter of the high schools and over 38 percent of the 
middle schools did not believe the district provided adequate resources. 
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Exhibit 4.15. School and District Respondent Perceptions on District Provision of 
Adequate Resources over the Past Five Years 



Source: HPSGP School and District Surveys (AIR) 





Strongly 

Disagree 


Disagree 


Agree 


Strongly 

Agree 


No 

Answer 


Over the past five years, the district actively provided adequate resources 


School Respondents, Overall (n = 106) 


7.6 


22.6 


44.3 


19.8 


5.7 


Elementary (n = 44) 


9.1 


18.2 


43.2 


22.7 


6.8 


Middle (n = 34) 


5.9 


32.4 


35.3 


20.6 


5.9 


High (n = 28) 


7.1 


17.9 


57.1 


14.3 


3.6 


District Respondents (n = 49) 


2.0 


8.2 


59.2 


30.6 


- 



It is important to clarify, however, some of the responses of the five districts that disagreed with 
the statement in Exhibit 4.15. One respondent indicated that the response was conditional on 
time, and that he would strongly agree that the district provided adequate resources in the last 
year and a half. Two respondents suggested that it was not necessarily that the resources were 
inadequate, but rather that HPSGP schools were not treated differently than other schools in the 
district - as one remarked, “They were not proactive with these schools. They didn’t focus on 
these schools any more than any other schools.” 

However, case study respondents in Year 1 valued targeted services and supports provided by 
their district, such as the assignment of strong and proven leadership and instructional staff. Yet, 
such targeted actions were cited for schools in only two case study districts. At about a third of 
the other sites, principals, teachers, and sometimes students felt as if they were at a resource 
disadvantage relative to other schools in the district. They would point to the physical plant 
(especially the library) as one indicator of this, as well as referring to frequent turnover in local 
leadership. 

Given that the HPSGP, like II/USP before it, focused on the lowest performing schools we 
wanted to explore whether districts prioritized these struggling schools in their allocation of 
resources. While we recognize that perceptions are not the best objective measure, respondent 
perceptions are shown in Exhibit 4.16 to complement some of actual personnel counts shown 
above and to more broadly measure differential resource level perceptions. 
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Exhibit 4.16. School and District Perceptions of the Degree of Support in 
Personnel and Non-Personnel Resources over the Past Five Years 



Source: HPSGP School and District Surveys (AIR) 



Over the past five years, do you believe that the centrai district has provided more, equal, or less support to your 
school in relation to other schoois in each of the foilowing areas? 




Less 

Support 


Equal 

Support 


More 

Support 


Don't Know 


No Answer 


Administrative or leadership staff 


School Respondents, Overall (n = 106) 


10.4 


60.4 


20.8 


2.8 


5.7 


Elementary (n = 44) 


11.4 


63.6 


13.6 


4.6 


6.8 


Middle (n = 34) 


5.9 


67.7 


17.7 


2.9 


5.9 


High (n = 28) 


14.3 


46.4 


35.7 


0.0 


3.6 


District Respondents (n = 49) 


2.0 


67.4 


26.5 


0.0 


4.1 


Instructional staff 


School Respondents, Overall (n = 106) 


14.2 


64.2 


12.3 


2.8 


6.6 


Elementary (n = 44) 


15.9 


63.6 


6.8 


4.6 


9.1 


Middle (n = 34) 


11.8 


70.6 


8.8 


2.9 


5.9 


High (n = 28) 


14.3 


57.1 


25.0 


0.0 


3.6 


District Respondents (n = 49) 


0.0 


63.3 


32.7 


0.0 


4.1 


Other support staff (e.g., teacher aides, custodians) 


School Respondents, Overall (n = 106) 


17.9 


66.0 


7.6 


1.9 


6.6 


Elementary (n = 44) 


15.9 


65.9 


6.8 


2.3 


9.1 


Middle (n = 34) 


20.6 


61.8 


8.8 


2.9 


5.9 


High (n = 28) 


17.9 


71.4 


7.1 


0.0 


3.6 


District Respondents (n = 49) 


0.0 


83.7 


12.2 


0.0 


4.1 


Equipment, suppiies, and materials 


School Respondents, Overall (n = 106) 


11.3 


62.3 


17.0 


2.8 


6.6 


Elementary (n = 44) 


15.9 


61.4 


11.4 


2.3 


9.1 


Middle (n = 34) 


5.9 


61.8 


20.6 


5.9 


5.9 


High (n = 28) 


10.7 


64.3 


21.4 


0.0 


3.6 


District Respondents (n = 49) 


0.0 


77.6 


18.4 


0.0 


4.1 



Exhibit 4.13 shows that between 60 and 66 percent of the respondents from HP Only schools 
believed that the school received equal district support in terms of administrative, instructional, 
and other support staff and equipment, supplies, and materials. While 21 percent of the school 
respondents believed the district provided more support in terms of administrative or leadership 
staff, just 12 percent believed they had more instructional staff support. High school respondents 
were statistically significantly more likely than elementary school respondents to report more 
support for administrative and instructional staff. About 18 percent of the HP Only school 
respondents perceived less district resources in the area of other support staff in relation to other 
schools in the district. With respect to non-personnel resources, an overall 17 percent believed 
their school received more equipment, supplies, and materials. With one exception, all district 
respondents felt that the district provided equal or more support across all four resource 
categories. 



Significant at the 5 percent confidence level. High school respondents also statistically significantly more likely 
than middle school respondents to report more instructional staff support (10 percent confidence level). 
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In summary, these pereeptions suggests that - despite their participation in a program targeting 
the lowest performing schools in the state - the majority of HP Only schools did not receive 
differential district support in terms of administrative, instructional and support staff and non- 
personnel resources. In addition, about 10 to 15 percent of school respondents believed they were 
receiving less in these areas. It is also interesting to note the differences between school and 
district respondent perspectives on some of these measures. For example, while 14 to 18 percent 
of school respondents felt they had fewer instructional and support staff than other schools, no 
surveyed district administrator believed this. While perceptions can vary and are often 
inaccurate, even when wrong they can affect such factors as school morale and the degree to 
which schools feel supported in what can be difficult task to improve academic performance. 

Role of the District in Personnel Issues 

As salaries and benefits make up 81 percent of total K-12 education expenditures, personnel are 
by far the most prominent school resource. This section explores the ways in which districts 
maintained and distributed staff, including fully credentialed teachers, to HP Only schools, and 
concludes with a brief review of district challenges in addressing personnel issues in lower 
performing schools 

District Support in Personnei Stabiiity 

While a majority of surveyed school and district respondents reported that their 
district fostered a stable school administration, about a quarter disagreed. 

The role of the district in leading efforts to place highly quality staff in HPSGP schools was 
explored in the Year 1 case studies. In response to the question of whether there were any 
specific efforts to allocate some of their strongest administrators and/or teachers to sites 
struggling the most academically, respondents from seven of the nine district offices involved in 
the case studies said they did not have such a program in place. An assistant superintendent 
conceded that while the district did not make a concerted effort to re-distribute teaching staff, 
which would “run into resistance from the union,” it prioritizes assigning new staff to lower 
performing schools and involves principals of lower performing schools in the recruitment 
process. In regard to school leadership, the respondent asserted, “I have the best principals I can 
find in my underperforming schools. I just put an excellent principal in one of our SAIT schools 
- she’s the best I’ve got.” Unfortunately, this degree of special attention followed by specific 
action was rarely mentioned by other district officials interviewed for this study (including both 
case study and survey districts). 

As we did not find much evidence of district practices to alleviate the turnover in the case study 
schools, we posed a survey question to assess whether districts helped maintain personnel 
stability across the larger sample of schools. There was an interesting degree of alignment 
between district and school survey respondents regarding district support in maintaining stable 
administration and instructional staff at HP Only schools. Across both school and district 
respondents, about a quarter did not believe that the district had assisted in fostering a stable 
school administration (Exhibit 4.17). Respondents from schools with two or more principals over 
the past five years were statistically significantly less likely to believe their district fostered a 
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Stable administration, than those from schools with one principal. In terms of instructional 
staff, about 28 percent of both respondent groups disagreed that the district supported stability. In 
explaining why they disagreed, district respondents generally noted that personnel attrition was a 
problem district- wide and not just for HPSGP schools. 

Exhibit 4.17 School and District Respondents’ Perceptions of District Support in 
Maintaining Stable School Personnel 



Source: HPSGP School and District Surveys (AIR) 



Over the past five years... 


Strongly 

Disagree 


Disagree 


Agree 


Strongiy 

Agree 


No 

Answer 


The district heiped maintain a stabie schooi administration in my school. 


School Respondents, Overall (n = 106) 


3.8 


19.8 


49.1 


20.8 


6.6 


Elementary (n = 44) 


0.0 


22.7 


43.2 


27.3 


6.8 


Middle (n = 34) 


5.9 


17.7 


47.1 


20.6 


8.8 


High (n = 28) 


7.1 


17.9 


60.7 


10.7 


3.6 


District Respondents (n = 49) 


4.1 


22.5 


30.6 


36.7 


6.1 


The district helped maintain a stabie teaching staff in my schooi. 


School Respondents, Overall (n = 106) 


6.6 


21.7 


52.8 


13.2 


6.6 


Elementary (n = 44) 


5.7 


15.9 


50.0 


15.9 


9.1 


Middle (n = 34) 


9.1 


29.4 


50.0 


11.8 


5.9 


High (n = 28) 


3.6 


21.4 


60.7 


10.7 


3.6 


District Respondents (n = 49) 


0.0 


28.6 


42.9 


22.5 


6.1 



Differential Policies for HPSGP and Lower Performing Schools 

The vast majority of surveyed districts did not report policies for preferential 

treatment in staffing lower performing schools. 

To further our understanding of the ways in which districts support staffing, the district survey 
asked respondents about district policies related to recruiting, retaining, and redistributing school 
administrators with demonstrated effectiveness and high quality teachers in lower performing 
schools. Similar to the case study findings, the majority of districts reported no preferential 
policies to foster retention or redistribution of high quality administrators or teachers in lower 
performing schools. However, about 20 percent of respondents said they did have such policies, 
and nearly a third of respondents said they had preferential recruiting policies for high quality 
teachers. 



About 68 percent of respondents from schools with two or more principals believed that the district fostered a 
stable administration, versus 89 percent of respondents from schools with one principal. This difference is 
statistically significant at the 5 percent confidence level. 
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Exhibit 4.18 Incidence of District Policies on Recruiting, Retaining, and 
Redistributing Effective School Administrators and High Quality Teachers in 
Lower Performing Schools (n = 49) 



Source: HPSGP District Surveys (AIR) 





Yes % 


No% 


Missing % 


Recruiting school administrators with demonstrated effectiveness 


20.4 


73.5 


6.1 


Retaining school administrators with demonstrated effectiveness 


18.4 


77.6 


4.1 


Redistributing school administrators with demonstrated effectiveness 


22.5 


71.4 


6.1 


Recruiting high quality teachers 


32.7 


59.2 


8.2 


Retaining high quality teachers 


18.4 


71.4 


10.2 


Redistributing high quality teachers 


12.2 


79.6 


8.2 



Five of the 49 surveyed districts without specific policies regarding effective school 
administrators contended that such strategies were not applicable, as all schools in the district 
were low performing, hence rendering preferential treatment moot. Others without policies noted 
that while there were no formal procedures, ad hoc processes were in place, or that district 
support through intensive training and competitive salaries made formal strategies unnecessary. 

Districts with teacher policies commonly cited financial incentives for attracting teachers, 
particularly for hard-to-staff subjects such as math and science. Only two districts, representing 
nine HP Only schools, reported that HPSGP schools received priority in staffing assignments. 
Two others described strong professional development activities as a means to recruit and retain. 
While the survey question asked about differential policies that target lower performing schools 
specifically, four of the district respondents acknowledged that the existing policies applied to all 
schools, while two others noted that they pertained to all schools, with a focus on lower 
performing schools. Similarly, respondents from 14 districts without differential polices for 
staffing teachers asserted that such steps were not applicable, due to reasons such as the district’s 
small size, all schools in the district were low performing, or all teachers were, by their 
definition, highly qualified. 

In a subsequent section, we present some of the confounding factors perceived by districts in 
establishing preferential policies for staffing. 

Raising the Percentage of Fully Credentialed Teachers in HPSGP Schools 

Although more than half of districts with HP Only schools fell short of the program 
assurance that these schools would be at least at the district average percentage of 
fully credentialed teacher, the degree to which they missed this objective on 
average was very small. 

Included in the HPSGP application for the first cohort were a series of six assurances detailing 
specific actions with which districts must comply as participants in the program. One of the 
assurances charges districts to ensure that by the second year of HPSGP implementation, the 
percentage of fully credentialed teachers in participating schools will increase at least to the 
district average. About 73 percent (n = 36) of the surveyed districts respondents were aware of 



The district assurance in the original CDE document, entitled: “Application Information for Schools Applying in 
October, 2003 to the High Priority Schools Grant Program,” reads: “No later than the end of the second year of 
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this HPSGP district assurance. Of those 36 respondents, 75 percent believed that they had met 
the target. 

Using CBEDS data, Exhibit 4.19 displays the percentage of districts in which all HP Only 
schools were at or above the average for their district regarding the percentage^** of fully 
credentialed staff between 1999-2000 and 2005-06. While we observe a steady increase in this 
percentage, less than half of the districts were in full compliance with this assurance by 2005-06. 
The exhibit also shows the percentage of HP Only schools that were at least at the district 
average percentage of fully credential teachers; more than 60 percent of HP Only schools were at 
or above the district average in the last year shown. It is important to note that the gap between 
the HP Only schools and the district average in districts that did not meet this assurance is fairly 
small. As depicted in Exhibit 4.20, in the aggregate, HP Only schools in these districts were 
under the district average by just 1.2 percent in 2004-05, which shrunk to less than one-half 
percent in 2005-06.^* 

Exhibit 4.19 Percentage of HP Oniy Schoois and HPSGP Districts in which aii HP Oniy 
Schoois were At or Above the District Average Percent of Fuiiy Credentiaied Teachers, 

1999-2000 to 2006-05 




implementation, the percentage of fully credentialed and experienced teachers will increase at least to the district 
average. The increase after the first year of implementation will be at least one half of the total increase needed.” 

® The district percentage reflects 94 districts which had at least one HP Only school and for which there were 
CBEDS data for all seven years. 

The district average of the percentage of teachers who are fully credentialed used in this analysis is a weighted 
district wide average, including all CBEDS school types, i.e., the total number of fully credentialed teachers in the 
district divided by the total number of teachers in the district). No major differences were seen in the results when 
using an unweighted average. 

Exhibit 4.20 groups districts by whether they were in compliance with this assurance by October 2004, two years 
after the implementation of the HPSGP. 
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Exhibit 4.20 Difference between the Percentage of Fully Credentialed Teachers in 
HP Only Schools and the District Average, in Districts Meeting and Not Meeting 
Compliance, 1999-2000 to 2005-06 




District Challenges in Supporting Personnel in Lower Performing Schools 

In describing reasons why there was a lack of targeted policies or limited district success in 
maintaining stable school-site personnel or increasing credentialed teachers, district respondents 
identified several challenges in recruiting and retaining personnel. Twenty percent of the 
surveyed districts with a general lack of preferential policies for lower performing schools 
reported that they had limited flexibility due to contractual and union issues. While 12 district 
respondents specifically reported no challenges in meeting the HPSGP district assurance relating 
to the percentage of fully credentialed teachers, 16 percent (n = 8) described the high level of 
teacher turnover as a confounding factor in reaching this goal. Conversely, five district 
respondents believed that turnover assisted the process, as it allowed the district to replace 
ineffective teachers. Contextual factors such as rural or remote location, challenging student 
population, or high cost of living made working in schools in ten districts unappealing, therefore 
hampering recruiting and retaining efforts. With a high percentage of highly qualified teachers in 
elementary schools, seven districts cited difficulties almost exclusively at the secondary level, 
particularly for math, science, and special education teaching positions. 
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Other Types of District Support 

Through the Year 1 case studies, we identified three key district practices that supported school 
improvement efforts: ongoing provision of student assessment data; professional development; 
and efforts to assign and maintain strong staff. However, because the prevalence of these 
practices were somewhat mixed in our limited case study sample, the school and district surveys 
attempted to gather information on these topics from a broader sample. As staff quality and 
stability were discussed above, this section reviews the results of survey items regarding district 
support in the areas of professional development and data systems on student achievement. 

Professional Development 

Professional development was cited as an important and sometimes effective role for the district 
by approximately half the teachers on average across all district sites participating in the Year 1 
case studies. While teachers cited its potential importance, some questioned the effectiveness of 
the district-level professional development. For example, internally or collaboratively provided 
professional development through local teacher collaboration was seen as more useful than one- 
day trainings from external providers. 

The school and district surveys included questions about the role of the district in the provision 
of professional development; results are shown in Exhibit 4.21. On average, about 72 percent of 
all school types believed that their districts provided quality professional development 
opportunities for principals and teachers. High school respondents held a more favorable view of 
professional development for principals, with 82 percent believing that the district provided 
quality opportunities in relation to about 68 percent of the other school types. While perceptions 
on opportunities for teachers was fairly consistent across schools types, district respondents were 
more likely to believe that the district provided quality professional development for teaching 
staff. 



Exhibit 4.21. School and District Respondent Perceptions on District Provision of 
Professional Development 

Source: HPSGP School and District Surveys (AIR) 



Over the past five years... 


Strongly 

Disagree 


Disagree 


Agree 


Strongiy 

Agree 


No 

Answer 


The district provided quaiity professionai deveiopment opportunities for principais. 


School Respondents, Overall (n = 106) 


2.8 


19.8 


45.3 


26.4 


5.7 


Elementary (n = 44) 


4.6 


20.5 


43.2 


25.0 


6.8 


Middle (n = 34) 


0.0 


26.5 


41.2 


26.5 


5.9 


High (n = 28) 


3.6 


10.7 


53.6 


28.6 


3.6 


District Respondents (n = 49) 


0.0 


16.3 


40.8 


38.8 


4.1 


The district provided quaiity professionai development opportunities for teachers. 


School Respondents, Overall (n = 106) 


0.9 


20.8 


47.2 


25.5 


5.7 


Elementary (n = 44) 


2.3 


18.2 


45.5 


27.3 


6.8 


Middle (n = 34) 


0.0 


20.6 


50.0 


23.5 


5.9 


High (n = 28) 


0.0 


25.0 


46.4 


25.0 


3.6 


District Respondents (n = 49) 


0.0 


6.1 


42.9 


51.0 


- 
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Data Systems 

Ongoing assessments and use of data were cited as important elements in improving school 
academic performance in about half of the 16 case study schools in Year 1, and was specifically 
referenced in four of the nine districts. Case study school respondents in four districts described 
the ongoing assessments as instrumental in assisting the schools to assess strengths and 
weaknesses of individual students, classes, grade levels, as well as the school as a whole. As an 
example of data support, one principal cited the helpfulness of an online database developed by 
the district’s research department. The system was said to provide easy access to school, teacher, 
and individual student records and to be increasing accountability at all levels including students 
being responsible for their own performance. School site members said that they had worked 
with a district research team “to change our culture around data-driven decision-making using 
the district data system to focus on nationally recognized instructional strategies.” Another site 
described the district as having changed its data policies to currently give quarterly tests to help 
teachers determine if they are teaching effectively. 



While all districts surveyed in Year 2 said that they currently had an on-going system in place to 
assess student performance, the extent to which the district provided useful support to schools in 
this domain over the past five years varied (Exhibit 4.22). About 71 percent of the surveyed 
school respondents and 86 of the districts believed that the district provided helpful on-going 
student achievement data and supported schools in data interpretation. Four of the five district 
respondents that disagreed noted that effective support from their district was only a recent trend. 
An overwhelming 84 percent of the surveyed district respondents were confident that the 
teachers were regularly using data to inform classroom instruction. 



Exhibit 4.22. School and District Respondent Perceptions on Student 
Achievement Data Systems 



Source: HPSGP School and District Surveys (AIR) 



Strongly 

Disagree 



Disagree 



Agree 



Strongly No 
Agree Answer 



Over the past five years, the district provided useful, on-going data on student achievement and assisted 
my school in interpreting the data. 



School Respondents, Overall (n = 106) 


2.8 


20.8 


38.7 


32.1 


5.7 


Elementary (n = 44) 


4.6 


18.2 


36.4 


34.1 


6.8 


Middle (n = 34) 


2.9 


23.5 


38.2 


29.4 


5.9 


High (n = 28) 


0.0 


21.4 


42.9 


32.1 


3.6 


District Respondents (n = 49) 


2.0 


10.2 


30.6 


55.1 


2.0 


Teachers in my district regularly use data to inform classroom instruction. 


District Respondents (n = 49) 


0.0 


14.3 


61.2 


22.5 


2.0 



Local School Boards 

Just as schools and districts can work together to raise student achievement, districts can also 
forge supportive relationships with local school boards. Local school boards can play an 
important role in education governance and in effecting change in schools. Conversely, as was 
observed in two of the 16 case study schools, these relationships may also be dysfunctional and 
disruptive to school reform. The following section provides a brief overview of district 
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respondents’ perceptions of the impact of their local school boards on school improvement 
efforts. 

Exhibit 4.23 presents the perceived impact of the school board on the improvement efforts of the 
schools in the sampled districts. Over two-thirds of the 49 district administrators surveyed 
indicated that their boards facilitate school improvement efforts, while a smaller percentage (20 
percent) reported they had no impact on school improvement efforts. While none characterized 
their boards as being a major impediment, 6 percent described the board as somewhat disruptive. 

Exhibit 4.23. District Respondents’ Perceptions of Impact of the Local School 
Board (n = 49) 



Source: HPSGP District Surveys (AIR) 





N 


Percent 


Facilitated school Improvement efforts 


33 


67.3 


No Impact on school Improvement efforts 


10 


20.4 


Somewhat disruptive 


3 


6.1 


A major impediment to school improvement efforts 


0 


0.0 


No response 


3 


6.1 



An interesting pattern is evident when comparing districts that described themselves has having 
clear cohesive goals and focused on school improvement and districts that reported they were in 
the process of developing school improvement goals. Suggestive of how school boards can 
contribute to an overall supportive environment, the vast majority of districts in the former group 
(88 percent) reported their board to have facilitated improvement efforts while less than half of 

72 

the latter group (46 percent) described the board in the same manner. 

School boards identified as having a positive influence were often directly involved in decision- 
making, supported the hiring of qualified personnel, were informed of issues and progress, 
and/or committed fiscal resources to schools to support reform efforts. Thirty percent of the 
districts that described the boards as facilitating improvement highlighted their focus on and 
expectations for student achievement. One district administrator recalled, “They set a district 
goal of improving test scores and put the pressure on us. It was good; we need that.” A few other 
districts cited the board’s willingness to head to a new direction, their attentiveness to 
recommendations and staff needs, and their resolve to hold schools and the district accountable 
as some examples of ways the board assist with improvement efforts. In one particular case, the 
board gave the superintendent wide discretion in appropriately filling positions in the district. 
Some respondents indicated that the board’s support was not always consistent. Five districts 
pointed out that their board assisted school improvement efforts for only part of the duration of 
the grant, primarily during the latter years. 

Among the ten district administrators that reported no impact, two stated that the board was 
generally supportive but their efforts had no bearing on improvement efforts while another two 
described the board’s involvement as fairly minimal. One district mentioned, “They don’t 
interact with us at all. We are so large that unless there is a major decision to start or stop 
something, they just approve the money from grants when it comes in.” While noting a high 



Significant at the 1 percent confidence level. 
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alignment between the board and the superintendent, another respondent said, “The local school 
board is not an entity that asks the tough questions.” 



Summary 

This chapter provided information on the school and district context in which the HPSGP was 
implemented, with a particular focus on personnel capacity and the degree of district support. 
Although personnel levels and quality in HP Only schools do not appear to substantially differ 
from other schools, the fact that these schools serve more academically challenging student 
populations suggests that equal resources may not be sufficient for these schools. To 
meaningfully assist HP Only schools, districts may need to actively provide differential levels of 
support. 

However, district support did not appear to be universal across the sample of HP Only schools, 
nor consistent across different types of support. The majority of HP Only school respondents did 
not report differential district support in terms of administrative, instructional, and support staff 
and non-personnel resources. In addition, the vast majority of surveyed districts reported no 
policies for preferential treatment in staffing lower performing schools. About a quarter of 
surveyed school and district respondents did not believe the district fostered a stable school 
administration or instructional staff. Survey evidence suggests that continuity of staff, 
particularly school leadership, is important for school reform coherence. For example, 
respondents in schools with two or more principals over the last five years were more than twice 
as likely to believe that the school improvement program lacked coherence in comparison to 
respondents in schools with one principal. 

These contextual factors may have influenced the degree to which schools implemented the 
HPSGP, as well as enhance our understanding as to why we observed negligible program effect 
on student achievement. When interpreting the findings of the following chapter, which 
examines the implementation of specific HPSGP components, the reader should be mindful of 
this broader context. 
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Chapter 5: Implementation of the HPSGP 



Introduction 

Fidelity of program implementation - or conversely, breakdowns in the process - can affect the 
potential of the HPSGP to impact student outcomes. The groundbreaking work of Berman and 
McLaughlin (1978) dealing with change in educational organizations describes the difficulties 
associated with introducing and implementing change programs in schools. Berman and 
McLaughlin studied 293 
American school change 
programs during the 1960s 
and 70s, and found that most 
of these programs, which had 
been developed by external 
bodies and “passed down” to 
the schools, were not well 
implemented at the local 
level. 

Despite the design efforts 
underlying HPSGP and the 
considerable resources 
associated with this program, 
there is a lack of 
educationally significant 
impact of the program on 
student achievement. This 
chapter attempts to explore 
the extent to which 
breakdowns in program 
implementation may be contributing to these findings. 

This chapter examines four fundamental elements of the implementation of the HPSGP: the 
development of the Action Plan with the assistance of an External Provider, district 
participation in the development, monitoring and implementation of the Action Plan, and the 
allocation and use of supplemental state funds. The findings presented in this chapter primarily 
draw upon the school survey results from HP Only schools, but are also informed by the district 
telephone surveys, the case study analyses conducted in Year 1 of this study, as well as other 
data sources such as the HPSGP expenditure files. 



Key Findings 

• Sixty percent of survey respondents reported that a 
plan for school improvement figured prominently in 
their reform efforts. However, this impression does 
not correlate with measured academic gains. 

• While nearly half of the school respondents 
described their external provider support as 
appropriate and effective, nearly 45 percent reported 
that the school did not use or did not know if the 
school used an external provider even though it was 
a program requirement. 

• Although the HPSGP attempted to more clearly define 
the district role, over 40 percent of school 
respondents indicated that the district had not 
actively assisted them in the implementation of the 
program. 

• The vast majority of HPSGP school respondents 
indicated an effective use of funds. However, half 
expressed concern about the number of years of the 
program, and nearly a third reported that the untimely 
arrival of funds did not allow appropriate planning 
and spending. 
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Before presenting the survey results pertaining to the HPSGP’s eomponents, eonsideration 
should be given to the respondents’ level of understanding of the HPSGP. While the survey was 
targeted to the most knowledgeable person at the school site,’^ it was possible that respondents 
would have limited program awareness, given the level of turnover we had observed in the case 
study schools. Despite concerns (given high turnover levels) that few or no personnel with 
program-specific experience or knowledge remained in our surveyed schools, the survey results 
indicate a high degree of familiarity with the HPSGP. Just one survey respondent at each school 
level (i.e., elementary, middle, high) indicated that they were only “somewhat” familiar with the 
program. Although respondents, on average, professed knowledge of the program, the reader 
should be mindful that the survey results reflect individuals’ perceptions, which may not 
accurately represent the views of the school overall. 

The following sections explore in greater detail respondents’ perspectives on the prominence of 
the Action Plan in school reform efforts, the appropriateness of the external provider, and the 
level of district HPSGP-related support. The chapter concludes with an in-depth examination of 
the HPSGP funding and expenditures, and discusses factors that limited schools’ ability to 
leverage their funds to improve student achievement. 



Action Plan 

Although more than 60 percent of all survey respondents reported that a plan for 
school improvement figured prominently in their reform efforts, this reported 
impression does not correlate with measured academic gains. 

Each school’s Action Plan lays out the framework for implementing change under the HPSGP. 
As described in the application, “The action plan will serve as a blueprint for the school and 
community to focus on raising student achievement to meet the school’s academic growth 
targets.” The Education Code (52055.620) provides specific guidelines for the development of 
this plan, which is to be research-based, include ongoing data gathering, and be grounded in the 
results from the initial needs assessment. With technical assistance from an outside expert 
knowledgeable about the challenges specific to low-performing schools (described further in the 
next section), HPSGP schools were required to address 14 dimensions of school improvement in 
the Action Plan. These dimensions included the identification of barriers to academic 
achievement at the school and district, specification of strategies to address these barriers, 
strategies to focus on literacy with an emphasis on English learners and other numerically 
significant subgroups, and plans for involving teachers in AB 466 training and administrators in 
AB 75 training. (Please see Technical Appendix G for the full list of overarching guidelines and 
specific plan requirements.) 

A general observation across the 16 case study schools visited during Year 1 of this evaluation 
was the lack of a current, distinct HPSGP Action Plan beyond the narrative that the school and/or 
district had submitted as part of the application process. While some respondents at nearly all of 
the case study site remembered some involvement in creating an HPSGP Action Plan, continuity 
of the plan was complicated by turnover - both among teachers and principals - at a majority of 



73 



Principals were encouraged to seek the assistance of another staff person who is knowledgeable about the HPSGP. 
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the visited schools. As a result, relatively few of the respondents at these sites reported that they 
considered the Action Plan as still meaningful in guiding current practice at the school. However, 
the Action Plan reportedly actively guided reform efforts in a third of the case study schools. 
Even in these schools, however, it was sometimes noted that while the plan was still adhered to, 
it was not evolving. 

At the same time, most of the case study schools did have some type of plan in place, particularly 
the Single Plan for Student Achievement (SPSA). Given this, and as the HPSGP schools were 
expected to integrate the HPSGP Action Plan with their SPSA, the school survey for this 
evaluation posed a broader question to assess the degree to which the school was guided by any 
plan for improvement, which could include the SPSA or the HPSGP Action Plan. 

Relative to the observations noted during the case studies, the survey results (shown in Exhibit 
5.1) suggest that the plan for school improvement had greater relevance across this much broader 
array of HPSGP schools. Nearly 70 percent of the school respondents asserted a high degree of 
knowledge of the school’s Action Plan (not shown in exhibit), and more than 60 percent 
described the plan as the “centerpiece for my school’s improvement efforts.” As described by 
one surveyed school respondent, “The Action Plan was the clout required to bring along all staff. 
The habits formed are invaluable to sustaining change at the school.” Eess than one-third 
indicated that the plan only partially guided their efforts. Pour schools did not have an active 
plan, while two schools reported that they did not have a clear plan. 



Exhibit 5.1 Percentages of School Respondents by the Degree to which the 
School is Guided by a Plan for Improvement 

Source: HPSGP School Surveys (AIR) 



Which of the following best describes the extent to which your school has been guided by a plan for 
improvement over the past five years?* 




This plan has been the 
centerpiece for my 
school’s improvement 
efforts and has actively 
guided all Instructional 
decisions. 


This plan has only 
partially guided my 
scheol’s 
Imprevement 
efforts. 


This plan has not 
played an active rele 
In guiding my 
school’s 

Improvement efforts. 


My school has 
not had a clear 
plan fer 
Improvement. 


School Respondents, 
Overall (n = 106) 


62.3 


32.1 


3.8 


1.9 


Elementary (n = 44) 


59.1 


31.8 


6.8 


2.3 


Middle (n = 34) 


64.7 


32.4 


2.9 


0.0 


High (n = 28) 


64.3 


32.1 


0.0 


3.6 



Source: HPSGP School Surveys (AIR) 

* This plan could be the school’s Single Plan for Student Achievement and/or the HPSGP Action Plan. 



Despite the perceived importance of the plan as shown above, this impression does not correlate 
with academic gains by the respondent schools. That is, schools whose respondents described the 
plan as a centerpiece of their improvement efforts performed no differently on average than those 
schools indicating much less (or no) reliance on such a plan.^"* Purthermore, the performance 



For this exercise, we examined the proportion of schools that met their 2006 API schoolwide and comparable 
targets, as well as their average performance on the CSX over the past five years (see Chapter 2 for further 
explanation of this grouping of higher and lower performers). No statistical differences were found between schools 
that described the plan as a centerpiece and other schools that did not describe the plan as a centerpiece. 
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levels of the group that perceived their plans as actively guiding their efforts were split fairly 
equally; roughly half of this group did not meet their 2006 API schoolwide and comparative 
growth targets, and the schools fell into nearly equal camps of higher and lower performers. This 
could suggest, among other things, that their perceptions of the plan may be overly confident, or 
that the plan does guide improvement efforts, but is not well-designed or sufficient for 
supporting gains in student achievement. 

In addition, the open-ended survey responses^^ for four schools indicate the need for additional 
guidance in devising a strategic plan for reform. While some school respondents valued the 
flexibility of the grant and the ability to individualize it to their schools’ needs, others expressed 
a desire to learn about and better understand effective practices. Two schools, in particular, noted 
that they were left to their own devices to identify areas for improvement and strategies: 

We were unable to ‘copy’ or use anyone else as a guide. The first few years we 
were left to ‘flounder’ on our own ... I think many of the schools in this first 
round of schools fumbled for a while trying to figure out what to do, how to do it, 
etc. My district did the best they could with the few people it has to provide 
assistance. 

I don’t feel we got the results we would have liked to have received. Once the 
grant was in operation, we learned from other schools some very innovative 
strategies that we would have liked to have known about before writing the grant. 

It might be a good idea if central office provides the schools with more 
suggestions or resources that are available. We do not want them to tell us what 
our school needs but we would like them to offer suggestions. 

The desire for clearer guidance and greater access to learning opportunities was similarly 
expressed by district respondents. These respondents indicated that they did not always have 
answers for the schools and pointed to the need for more technical assistance to come from the 
state. About 20 percent of the 49 surveyed districts identified the limited knowledge available 
and guidance on effective school reforms as a weakness of the HPSGP. A generally reported 
sentiment was that while the intention of the program was clear, the way to reach the goals was 
not. For example, one district respondent noted, “There is always the need to clarify strategies. I 
know there is no silver bullet, but I wish there was more information available; a small 
clearinghouse to know where to go to see what is good. We’ve had a leg-up, but someone 
coming in cold might be lost at where to start and what to do.” 



In response to the question, “What is the single most important improvement to the HPSGP that you would 
recommend to state policymakers?” 
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External Provider 

While nearly half of the school respondents described their external provider 
support as appropriate and effective, 44 percent of the surveyed respondents 
reported that^e school did not use or did not know if the school used an external 
provider in the development of the Action Plan, even though it was a program 
requirement 

Under the overall HPSGP design, the research expertise and guidance in regard to best practice 
comes from an external provider, who at a minimum was expected to guide the school in the 
formulation of its Action Plan. Questions addressed in this section include the various types of 
entities serving in an external provider capacity, over what period of time, and with what degree 
of follow-up, as well as the extent to which schools perceived this assistance to be appropriate 
and effective. 

In a significant departure from its precursor, the II/USP, the HPSGP allows entities other than 
state- approved External Evaluators to provide technical assistance in the development of the 
Action Plan. These other permitted entities for the first cohort of HPSGP schools included school 
district personnel, county offices of education, universities, or any other person or entity that has 
proven successful expertise specific to the challenges inherent in low-performing schools. 

Our case studies in Year 1 found considerable confusion regarding the external provider, and 
relatively little in the way of positive comments that this had been a helpful component of the 
school’s reform efforts. Knowledge of the external providers was either non-existent or negative 
across the vast majority of the 16 schools visited through this study. While a somewhat positive 
experience with the external provider was described at two sites, principal and teacher 
respondents from the remaining 14 sites were generally unaware that there had been such a 
person associated with their school, or generally reported that this person had not been helpful. 

Under the HPSGP, the external provider only needs to be involved in the first year, and then may 
or may not be continued. This role may be played internally, e.g., by the district office, and in 
those cases, the differential support appeared less clear cut. Eor one site, we interviewed the 
person listed on the application as the district external provider, who had no knowledge of this 
designation. Along with the considerable turnover in administrative and teaching staff, this may 
contribute to the general lack of knowledge observed across some of the visited schools. 

The survey built on these case study findings to determine the level of provider support and 
perceptions of their effectiveness across a wider sample of HPSGP schools and districts. Relative 
to other program components, school respondents generally had a lower comprehension of the 
role and responsibilities of the external provider, with less than half of the surveyed schools 
describing their knowledge as thorough (see Technical Appendix E). Perhaps the more striking 
finding from the survey regarding external providers is that only 54 percent of the school 
respondents reported that their school had used an external provider to develop an Action Plan 
for the HPSGP application. This leaves 44 percent of responding schools that either did not use 



In the second cohort of HPSGP schools, the district was no longer permitted to serve as the external provider. 
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(34.0 percent) or did not know if the school used an external provider (10.4 percent), which is 
quite surprising given that this was a program requirement. One explanation may be that the 
district served in the role of the provider, but - as found in our case studies - that the schools 
were unaware that the district was performing this function. Alternatively, it could relate to the 
lack of familiarity the respondent had regarding this program component. Respondents who 
reported that the schools used an external provider in the development of the plan were 
significantly more likely that have a thorough understanding of the roles and responsibility of the 
provider than those who reported no use or did not know. While the degree of administrative 
turnover may influence some principal respondents’ familiarity, principals were encouraged to 

7Q 

draw upon other staff who were familiar with the program in completing the survey. 

Among those schools reporting use of a provider, the majority used either a private company or 
individual, as observed in Exhibit 5.2. There was more variation in the use of district and county 
offices of education (COE) as providers. Middle schools appear most likely to utilize the COE, 
while only elementary schools reported using the district office. It is important to note that four 
schools identified a provider type, even though they reported that the provider was not involved 
in the development of the Action Plan. This suggests that some schools tapped into external 
sources to assist with the implementation of the plan after its development. 



Exhibit 5.2. Types of External Provider Among School Respondents Reporting 
Use of a Provider 

Source: HPSGP School Surveys (AIR) 





District Office 


County Office 
of Education 


Private 

Company 


Private 

Individual 


School Respondents, Overall (n = 61)* 


6.6 


21.3 


55.7 


16.4 


Elementary (n = 26) 


15.4 


15.4 


57.7 


11.5 


Middle (n = 20) 


0.0 


30.0 


50.0 


20.0 


High (n = 15) 


0.0 


20.0 


60.0 


20.0 



NOTE: “University” and “Other” were provided as options in the survey, but these were not reported. 

* One high school did not provide a response. The total number of respondents for this survey item (n = 61 ) is slightly larger than the 
total number of schools that responded “Yes” to having an external provider help develop the action plan (n = 57). Four schools that 
responded “No” in the earlier table later identified an external provider type. One school that answered “Yes” did not provide a 
response. 



Although the HPSGP does not require ongoing support from the external provider beyond the 
development of the plan, we were interested to what extent a longer term relationship was 
established. While schools working with an external provider reported an average of three years 
of involvement, the nature of that support appears to vary as shown in Exhibit 5.3. Elementary 
schools appear relatively less likely to have regular, ongoing support from their providers. 



As CDE does not maintain electronic records of entities serving as external providers, we were unable to test 
whether districts were the external provider in schools that reported no use of a provider, or did not know of this. 

About a quarter of the respondents who indicated that the school did not use a provider in the development of the 
Action Plan or did not know reported a thorough understanding of the roles and responsibilities of the external 
provider under the program, in relation to 63 percent of the respondents who indicated that the schools did use a 
provider. This difference is significant at the 1 percent confidence level. 

There were no statistically significant differences in how principals who had been at the school site for less than 
five years answered whether the school used an external provider, in comparison to those who had been at the site 
for five or more years. 
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whereas more than half of the middle schools reported a sustained relationship with regular 
interaction. 

Exhibit 5.3. Percentages of Schooi Respondents Reporting the Use of a Provider 
by Level of Support beyond the Development of the Action Plan 

Source: HPSGP School Surveys (AIR) 



Which statement best describes the support your school received from the external provider after the 
development of an Action Plan (e.g., through an additional contract or arrangement)? 





After the Action Plan 
was approved, our 
involvement with the 
external previder 
ended. 


The external 
provider provided 
limited follew-uD 
assistance after the 
development of the 
Action Plan. 


The external provider 
visited our school 
reaularlv to helo us 
implement the Action 
Plan. 


School Respondents, Overall (n = 61)* 


13.1 


42.6 


44.3 


Elementary (n = 26) 


19.2 


42.3 


38.5 


Middle (n = 19) 


5.3 


42.1 


52.6 


High (n = 16) 


12.5 


43.8 


43.8 



* This is based on school respondents who identified a provider type (n = 61). One middle school did not provide a response. The 
total number of respondents for this survey item (n = 61) is slightly larger than the total number of schools that responded “Yes” to 
having an external provider help develop the action plan (n = 57). Four schools that responded “No” in the earlier table later 
identified an external provider type. 



The survey also prompted respondents (who reported using a provider) to indicate their level of 
agreement on a series of statements pertaining to the appropriateness of the external provider 
support; these results are presented in Exhibit 5.4. On average, school respondents held favorable 
views of them, with three-fourths or more of the 61 school respondents that identified a provider 
type (as shown Exhibit 5.2) agreeing or strongly agreeing that provider was critical to the 
development of the Action Plan; provided effective, appropriate support; understood the school 
context; and established rapport with school staff. However, school-level analyses reveal striking 
differences, with elementary school respondents more than twice (and in one case, five times) as 
likely to disagree on these points. The differences between school types are smaller on the issue 
of whether the provider spent adequate time in the school, with 70 percent overall agreeing. 
Nearly two-thirds of all respondents believed that the provider was a critical factor in improving 
student achievement, and 80 percent overall would recommend their provider to other schools. 
Again, however, the overall average masks substantial variation at the school level, as 
elementary school respondents were more than three times as likely not to recommend their 
provider. 
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Exhibit 5.4. Percentages of Survey Respondents by their Perceptions of the 
Appropriateness of Externai Provider Support 



Source: HPSGP School and District Surveys (AIR) 





Strongly 

disagree 


Disagree 


Agree 


Strongly 

agree 


No 

answer 


The external provider.... 












a) Was critical to the development of the school’s Action Plan. 










School Respondents, Overall (n = 61) 


1.6 


24.6 


49.2 


23.0 


1.6 


Elementary (n = 26) 


3.9 


38.5 


30.8 


26.9 


- 


Middle (n = 20) 


0.0 


15.0 


60.0 


20.0 


5.0 


High (n = 15) 


0.0 


13.3 


66.7 


20.0 


- 


b) Provided effective, appropriate support for my school. 


School Respondents, Overall (n = 61) 


6.6 


8.2 


62.3 


19.7 


3.3 


Elementary (n = 26) 


11.5 


11.5 


53.9 


23.1 


- 


Middle (n = 20) 


5.0 


5.0 


65.0 


15.0 


10.0 


High (n = 15) 


0.0 


6.7 


73.3 


20.0 


- 


District Respondents (n = 39) 


0.0 


20.5 


33.3 


33.3 


12.8 


c) Understood issues faced by our school. 


School Respondents, Overall (n = 61) 


3.3 


9.8 


59.0 


26.2 


1.6 


Elementary (n = 26) 


3.9 


23.1 


50.0 


23.1 


- 


Middle (n = 20) 


5.0 


0.0 


65.0 


25.0 


5.0 


High (n = 15) 


0.0 


0.0 


66.7 


33.3 


- 


d) Spent adequate time at our school to learn about our needs and challenges. 


School Respondents, Overall (n = 61) 


6.6 


21.3 


55.7 


14.8 


1.6 


Elementary (n = 26) 


11.5 


23.1 


50.0 


15.4 


- 


Middle (n = 20) 


5.0 


20.0 


55.0 


15.0 


5.0 


High (n = 15) 


0.0 


20.0 


66.7 


13.3 


- 


e) Established a good rapport with our faculty and staff. 


School Respondents, Overall (n = 61) 


8.2 


13.1 


52.5 


23.0 


3.3 


Elementary (n = 26) 


11.5 


19.2 


46.2 


23.1 


- 


Middle (n = 20) 


5.0 


10.0 


55.0 


20.0 


10.0 


High (n = 15) 


6.7 


6.7 


60.0 


26.7 


- 


f) Was a critical factor in improving student achievement at my school. 


School Respondents, Overall (n = 61) 


8.2 


23.0 


54.1 


11.5 


3.3 


Elementary (n = 26) 


11.5 


26.9 


46.2 


15.4 


- 


Middle (n = 20) 


5.0 


15.0 


65.0 


5.0 


10.0 


High (n = 15) 


6.7 


26.7 


53.3 


13.3 


- 


District Respondents (n = 39) 


2.6 


28.2 


28.2 


30.8 


10.3 


g) 1 would recommend this external provider to other schools. 


School Respondents, Overall (n = 61) 


3.3 


14.8 


50.8 


29.5 


1.6 


Elementary (n = 26) 


3.9 


26.9 


38.5 


30.8 


- 


Middle (n = 20) 


5.0 


5.0 


55.0 


30.0 


5.0 


High (n = 15) 


0.0 


6.7 


66.7 


26.7 


- 



NOTE: These statements were analyzed only for schools that identified an external provider type (n = 61). The total number of 
respondents for these survey items (n = 61) is slightly larger than the total number of schools that responded “Yes” to having an 
external provider help develop the action plan (n = 57). Four schools that responded “No” in the earlier table later identified an 
external provider type. 



School respondents that reported the use of an external provider in the development of the 
Action Plan were statistically significantly more likely to describe their plan as actively guiding 
school improvement efforts than respondents that did not know or did not use a provider (75 
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percent to 47 percent). However, given the tenuous link between the Action Plan and student 
achievement in our surveyed schools, this finding should be interpreted with some caution. 
Furthermore, there were no statistically significant differences in academic growth between 
schools whose respondents reported effective support from their provider and all other surveyed 
schools. 

District telephone survey respondents who reported an awareness of their schools’ external 
providers were also asked their impressions regarding two of the questions shown in the exhibit 
above. While two-thirds of the 39 district respondents who were fa mi liar with the external 
providers generally agreed that the providers’ support was effective and appropriate for HPSGP 
schools in their district, 20 percent indicated that this was not the case. Nearly 60 percent of the 
39 district respondents indicated that they had found their external providers to be a critical 
factor in improving student achievement. (However, the 12 district respondents reporting that the 
external provider was not a critical factor represent nearly one-third (126) of HP Only schools.) 

In districts that perceived the providers to be effective or a critical factor, respondents described 
the providers as knowledgeable and helpful in providing direction by identifying critical areas of 
need, building data capacity, and coaching. Four districts pointed to evidence of the ongoing 
impact of the providers’ support. Commending the provider’s comprehensive approach to staff 
development, one district noted, “Years later, we still see these activities happening in the 
school. They weren’t a passing fad; teachers saw the value in them.” The following quote from a 
district respondent provides one example of how the effective engagement of an external 
provider was perceived as a benefit to improvement efforts: 

Some of our principals are very new and as the external providers were well- 
versed and veteran educators, they could help the principal see some of the 
pitfalls they were getting into, identify effective practices, demonstrate classroom 
walk-throughs. The external providers provided a coaching role. They were there 
consistently (about 3 times a month) and were committed to the schools. 

District respondents who disagreed with the usefulness of the external providers did not seem to 
contest the value of external expertise, but rather questioned the quality and actions of particular 
providers. In providing reasons for the low opinion of the provider, one district administrator 
described the experience as demoralizing, because the provider blamed the staff and did not 
recognize the challenges of working in a very rural and impoverished area. Three districts were 
critical of the disconnect with district goals and efforts, with one noting, “It ended up making the 
school more dysfunctional because they were following what the provider was saying rather than 
working collaboratively [with the district].” 

In a district with six HPSGP schools, the respondent reported general consensus among their 
participating schools that the providers were not worth the investment: “All our schools were 
complaining about paying somebody $20,000 to come in three times a year, shake their hand, 
and say, ‘Hey, how are things going?’ They really had problems with the providers because they 
didn’t see them, and they didn’t really get anything in return except paperwork when reporting 



*** “All other schools” include schools whose respondents did not report effective support and schools whose 
respondents reported no provider or did not know if a provider was used. 
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time came.” Another district respondent lamented the lack of longevity in the reforms and 
quality control, noting that the competent providers already had full caseloads which opened the 
doors for people who had not been previously approved. 

In summary, the survey responses regarding the value of external providers appear somewhat 
more positive than what was reported through the case study analysis. This discrepancy may be 
due to different respondents (with the case studies relying on responses from a broad array of 
participants as opposed to school surveys completed primarily by principals) as well as the much 
larger number of survey respondents. However, the fact that nearly half of the survey 
respondents indicated that they had not used or did not know about the school’s external provider 
likely contributes to the difference in positive perceptions. As the use of an external provider was 

o 1 

a program requirement, it likely suggests that the provider did not play a prominent role. 

Where the provider had an impact and was successful, the memory may linger longer than in the 
case of one providing minimal and perhaps largely ineffective services. 

At best, these results regarding the perceived usefulness of external provider assistance appear 
quite mixed. Clearly for some schools, the external provider was perceived as effective. These 
findings seem to suggest that an external provider to the school reform process could be quite 
helpful when a good one could be found. It seems this was not always easy, as articulated by 
respondents who expressed a need to know more about how to best select the providers. As one 
asserted, “I think we would have gotten where we are now much faster if we had a plan and were 
given some advice about who were the external providers who could get the job done.” A school 
respondent in a different district corroborated this response, “We used a total of three different 
external evaluators/consultants, which caused a two year delay in determining and supporting the 
most effective course of action.” 



District Support under the HPSGP 

Although the HPSGP attempted to more clearly define the district role, over 40 
percent of school respondents indicated that the district had not actively assisted 
them in the implementation of the program. 

In another change from the II/USP, the HPSGP more clearly defined the district’s role in school 
improvement by requiring their participation in the development of the Action Plan and a district 
monitoring component. According to the program application guidelines, after approval of the 
Action Plan developed in partnership with the school site council, the school and district 
administration are responsible for its timely and effective implementation. Participating districts 
are also required to maintain all fiscal records and submit annual data on various elements of the 
program. 

While earlier in this report we presented survey results on the wider district context during the 
grant period, here we focus on the district’s support specifically in relation to the program: 



** One argument might be that the respondents who reported that the school did not use an external provider in the 
development of the Action Plan or that they did not know were not present at the start of the program. However, the 
responses of respondents who had been at the school for five or more years were similar to the overall average (56 
percent said the school used an external provider, 33 percent said no, and 8 percent did not know). 
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development of the Action Plan and program implementation (see Exhibit 5.5). More than two- 
thirds of the surveyed schools believed that the district provided sufficient support in developing 
the Action Plan, whereas a smaller percentage (57 percent) agreed that the district actively 
assisted in the implementation of the program. High school respondents were more likely than 
other school types to have a favorable view of the district’s support in the planning, but less 
likely in implementation. However, there were no statistically significant differences in academic 
growth between schools whose respondents reported that the district actively assisted in program 
implementation and schools whose respondents did not report this support. 



Exhibit 5.5. Percentages of Survey Respondents by their Perceptions of District 
Support Specific to the HPSGP 



Source: HPSGP School Surveys (AIR) 





Strongly 








No 




disagree 


Disagree 


Agree 


Strongly agree 


answer 


The central district provided sufficient support in the development of the HPSGP Action Plan. 


School Respondents, Overall (n = 1 06) 


3.8 


24.5 


51.9 


12.3 


7.6 


Elementary (n = 44) 


9.1 


22.7 


40.9 


15.9 


11.4 


Middle (n = 34) 


0.0 


35.3 


52.9 


5.9 


5.9 


High (n = 28) 


0.0 


14.3 


67.9 


14.3 


3.6 


The central district actively assisted in the implementation of the HPSGP. 


School Respondents, Overall (n = 1 06) 


6.6 


32.1 


41.5 


13.2 


6.6 


Elementary (n = 44) 


11.4 


22.7 


40.9 


15.9 


9.1 


Middle (n = 34) 


5.9 


35.3 


47.1 


5.9 


5.9 


High (n = 28) 


0.0 


42.9 


35.7 


17.9 


3.6 



Note: Charter school responses were re-coded as “missing.” 



Although these data indicate that a majority of school respondents found district support under 
the HPSGP to be generally sufficient, the perceived lack of sufficient district support reported by 
30 percent of school respondents in regard to development of the Action Plan and 40 percent in 
regard to HPSGP implementation suggest an important source of implementation breakdown in 
these schools. 

While districts are charged with maintaining fiscal records and - along with the school - 
ensuring the program’ s implementation, some school respondents leveled criticism at how the 
districts fulfilled these duties. As discussed in more detail later in this chapter, eight surveyed 
schools explicitly identified the district as a constraining factor in using the HPSGP funds to 
effect change. Evidence from our case studies and surveys indicate that the degree of district 
involvement in regards to monitoring and oversight varied substantially. Some districts acted as 
strong central authorities, meticulously approving and denying funds for very specific purposes, 
while others were more hands-off allowing schools more flexibility in their spending. An 
indicator of district involvement, as of June 2007, nearly 60 percent of the districts with HP Only 
schools still had not submitted 2005-06 data for all of their HP Only schools, which resulted in 
missing data for well over a third of these schools. 
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Among the eight surveyed district respondents who noted the need for more state guidance when 
asked about their most pressing concerns in regard to HPSGP implementation, two explicitly 
mentioned a lack of clarity around district responsibilities under this program. As one described, 
“We did not know the expectations until well after the grants were implemented in terms of what 
were the expectations, monitoring, or exiting [criteria]. Everything came after the fact and not 
part of the original grant application process.” Confusion regarding the district role, however, 
may have been an artifact of moving from the lEUSP to the HPSGP, or trying to support schools 
in both programs with different requirements. The other district respondent also noted the 
absence of follow-up regarding the annual reports, “We’ve submitted reports for four years now, 
and there was never any feedback. We’re not sure if we’re doing the right thing or if we don’t 
receive feedback because we’re doing okay. So we try to base it on ourselves whether things are 
working or not.” While feedback on the Annual Report is not required by the HPSGP legislation, 
this comment nonetheless points to a district desire for more communication and guidance from 
the state. 

Concerns regarding district support of school reform efforts were a theme of both evaluations of 
the II/USP, the predecessor to the HPSGP, perhaps leading to the more specific delineation of 
the district role and responsibilities in regard to the HPSGP. However, uniformity of district 
support appears to be a continuing problem in regard to the state’s school reform efforts. One 
surveyed school respondent, who strongly disagreed that the district actively assisted in the 
implementation of the program, emphasized the need for district support: 

School improvement is only as good as the people carrying out the reform. 

Funding is a critical part of that picture, but it is not the only part. A lot of money 
given without supports in place to help the people who are charged with carrying 
out the plan for improvement is bound to fail. Having experience the reform at 
my school for the past five years, I have found that the support of the district in 
the reform movement is critical. The district must be actively involved in the 
reform that is expected to happen at the school site level. Without a deep 
knowledge of the plan for each school, the district can get in the way of the plan 
being implemented the way it is intended. 



HPSGP Funding 

This section covers overall funding issues and spending patterns for the HP Only schools 
participating in the Year 1 survey, drawing also upon data on expenditures for all HP Only 
schools. It also explores the degree to which HP Only schools carried over unspent program 
funds, and concludes with an examination of annual available funding per student given the 
carry-over and changes in school enrollment. 
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Funding Overview 

Although the vast majority of HPSGP school respondents indicated an effective use 
of funds, half expressed concern about the number of years of the program, and 
nearly a third reported that the untimely arrival of funds did not permit appropriate 
planning and spending. 

The infusion of fiscal resources is a significant component of the HPSGP, allowing schools to 
implement the Action Plan and recommendations made by the external providers. As described 
in the report’s introduction, participating schools were provided $400 per student in 
implementation funds based on their 2000-01 enrollments.*^ Exhibit 5.6 below displays the total, 
per school, and per student amounts awarded each year. Over the course of four implementation 
years, the state awarded a total of $542 million dollars to HP Only schools, averaging $1.6 
million per school and $1,511 per student.*^ 

Exhibit 5.6. Total Money Awarded to HP Only Schools, 2002-03 to 2005-06 



Source: CDE HPSGP Expenditure Files 2002-03 through 2005-06. 





2002-03 
(n = 338) 


2003-04 
(n = 349) 


2004-05 
(n = 349) 


2005-06 
(n = 347) 


Four Year 
Total 


Total Funds to all HP Only Schools 
Average Per School* 

Average Per Student* 


$113,567,920 

$336,000 

$310 


$146,779,280 

$434,258 

$397 


$144,687,488 

$428,069 

$402 


$136,760,560 

$404,617 

$401 


$541 ,795,248 
$1,602,945 
$1,511 



* Year-specific enrollments for schools with non-missing values from the expenditure files were used to calculate these values. In 
2002-03, schools were provided with 80 percent of the full implementation amount of $400 per pupil. This is why Exhibit 5.1 displays 
an average per pupil award amount of $310 - approximately 80 percent of the full $400 amount. 



The vast majority of respondents participating in the school survey indicated that the HPSGP 
funding amount was appropriate, used effectively and gave them access to new opportunities 
(Exhibit 5.7, rows a-c). However, other factors, namely a limited spending timeframe, 
uncertainty in the delivery of funds and unclear instructions around funding provisions appear to 
have tempered the potential impact of the funds in approximately a third of the responding 
schools (Exhibit 5.7, rows d-f). About half the school respondents disagreed with the statement 
that the longevity of the funds aligned well with improvement efforts, while approximately one- 
third disagreed that the funding was timely and the provisions clear. 

Eor the majority of these items there was consistency across school types. However, in 
comparison to the overall average, a greater percentage of high schools considered the funds 
timely and a smaller percentage of elementary schools perceived the funding provisions to be 
clear. The fact that 93 percent of the surveyed high schools versus 45 and 63 percent of the 
elementary and middle schools, respectively, are on traditional calendars may help explain these 
trends - evidence presented in the following sections suggests that school calendar and the 
perception of funding timeliness may be related. 



In 2002-03, schools were provided with 80 percent of the full implementation amount of $400 per pupil. This is 
why Exhibit 5.1 displays an average per pupil award amount of $310 - approximately 80 percent of the full $400 
amount. 

** After the 2004-05 school year, seven HP Only schools were identified for SAIT. These schools received the final 
20 percent of the HPSGP funds in 2005-06. 
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District respondents were less confident than school respondents about the effectiveness of the 
use of funds - 70 percent of district respondents compared to 98 percent of school respondents 
agreed that the funds were used effectively. When questioned about the timing of the funds, 
about 60 percent of district respondents acknowledged that the funds were received in a timely 
manner. However, about a quarter of district respondents supported school claims that they 
experienced some level of delay or uncertainty with the delivery of funds. 

Exhibit 5.7. School Perspectives on Funding under the HPSGP 



Source: HPSGP School Surveys (AIR) 





Strongly disagree / 
Disagree (%) 


Agree/ Strongly 
Agree 
(%) 


No answer 
(%) 


a) The HPSGP funds permitted the school to invest in strategies that otherwise wouid not have been 
possibie. 


School Respondents, Overall (n = 106) 


4.7 


95.3 


- 


Elementary (n = 44) 


4.5 


95.5 


- 


Middle (n = 34) 


5.9 


94.1 


- 


High (n = 28) 


3.6 


96.4 


- 



b) The HPSGP funds were used effectively in my school 



School Respondents, Overall (n = 106) 
Elementary (n = 44) 

Middle (n = 34) 

High (n = 28) 


1.9 
2.3 

2.9 
0.0 


98.1 
97.7 

97.1 
100.0 


- 


c) The amount of money received through the HPSGP seemed appropriate for my school's improvement 


efforts 








School Respondents, Overall (n = 106) 


13.2 


86.8 


- 


Elementary (n = 44) 


11.4 


88.6 


- 


Middle (n = 34) 


11.8 


88.2 


- 


High (n = 28) 


17.9 


82.1 


- 


d) The number of years of funding under the HPSGP seemed appropriate for my school's improvement 


efforts. 








School Respondents, Overall (n = 106) 


48.1 


51.9 


- 


Elementary (n = 44) 


50.0 


50.0 


- 


Middle (n = 34) 


44.1 


55.9 


- 


High (n = 28) 


50.0 


50.0 


- 


e) We received the HPSGP funds in a timely manner that enabled appropriate planning and spending. 


School Respondents, Overall (n = 106) 


34.9 


64.2 


0.9 


Elementary (n = 44) 


38.6 


61.4 


- 


Middle (n = 34) 


38.2 


61.8 


- 


High (n = 28) 


25.0 


71.4 


3.6 


f) The HPSGP funding provisions (e.g. carry-over) were clear. 


School Respondents, Overall (n = 106) 


31.1 


67.9 


0.9 


Elementary (n = 44) 


34.1 


63.6 


2.3 


Middle (n = 34) 


29.4 


70.6 


- 


High (n = 28) 


28.6 


71.4 


- 
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Carry-Over of Funds 

Approximately one-quarter of HPSGP funds program-wide were carried over each of 
the four implementation years. 

Under the HPSGP, schools were allowed to carry over unexpended grant funds. According to 
guidelines in the HPSGP application, carry-over was allowable for 2002-03 through 2004-05 
only. In 2005-06, the final year of the grant, carry-over was not allowed. However, late in the 
2005-06 school year, the CDE rescinded this policy. 

Exhibit 5.6 shows that as of the end of 2005-06, a total of $55 million in HPSGP funds remained 
unused, averaging $158,458 per school and $161 per student. Due to previous carry-over 
amounts, schools were carrying over increasingly more each year of the program. By the end of 
Year 4, approximately 8 out of every 10 HP Only schools had spent less of the grant than they 
received. Schools used 73 percent of the available HPSGP money and carried over 27 percent 
over the duration of the grant. By 2005-06, due to the accumulation of unspent funds from prior 
years, the typical HP Only school had at their disposal the equivalent of one and a half times the 
value of their annual award. 

Exhibit 5.8. Total Money Carried Over by HP Only Schools, 2002-03 to 2005-06 



Source: CDE HPSGP Expenditure Files 2002-03 through 2005-06. 





2002-03 
(n = 338) 


2003-04 
(n = 348) 


2004-05 
(n = 348) 


2005-06 
(n = 347) 


Total Carry-Over across all HP Only Schools 


$28,792,182 


$47,590,252 


$52,622,548 


$54,985,024 


Average Per School 


$85,184 


$136,754 


$151,214 


$158,458 


Average Per Student 


$79 


$129 


$147 


$161 



* Year-specific enrollments for schools with non-missing values from the expenditure files were used to calculate these values. 



Exhibit 5.9 below displays the distribution of HP Only schools by the percentage range of 
available funds carried over during the implementation period. The distribution in Year 1 (2002- 
03) is striking in comparison to subsequent years given the relatively higher percentage of 
schools with carry-over levels between 75 - 100 percent of their award (26 percent versus 3 
percent in following years). As it turns out, 80 percent of the schools in the largest carry-over 
category in Year 1 were schools that applied late and were funded towards the end of the 2002- 
03 school year. Given that most could not spend the money until Year 2, schools receiving the 
second round of funding reported carry-over levels near or at 100 percent of their available 
funds. If you consider only schools funded with first round funding in Year 1, three quarters 
carried over less than 25 percent of their available funds. 
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Exhibit 5.9. Distribution of HP Only Schools by Percent of Available Funds Carried 
Over, 2002-03 to 2005-06 (N = 351) 

Source: CDE HPSGP Expenditure Files 2002-03 through 2005-06. 
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23% 



7% 



32% 



26% 



6% 



2002-03 



4% 



8% 



33% 



40% 




3% 



12% 



30% 



39% 



9% 


7% 


5% 


5% 


8% 


9% 


2003-04 


2004-05 


2005-06 



Percent of 

Available 

Funds 

Carried Over 

■ 75-100% 

■ 50-75 % 

■ 25-50 % 

■ 0-25 % 

“ No Carryover 
□ Overspent 



1 



2 



Year 



3 



4 



Over time, the percentage of schools without carry-over or carrying over less than 25 percent of 
their available funds declines from 58 to 44 percent. The percentage of schools carrying over 
more than 25 percent of their funds increases after the first year from 32 to 46 percent, but 
remains more or less constant in subsequent years. Only six HP Only schools fully spent the 
entire value of the HPSGP award each year of the program. 

These observations raise important questions about the underlying conditions influencing how 
schools used the money: was the HP funding amount commensurate with the real needs of HP 
schools? Were there structural or systemic factors associated with program implementation that 
prevented HP schools from maximizing the use of the money? Was carry-over a strategic 
decision that benefited the school in some way? And finally, was carry-over an indicator of an 
incoherent spending plan among HP schools? These questions will be explored further in the 
following section that assesses factors limiting the use of the HPSGP funds — in many cases - 
carry-over was a meaningful indicator of deeper issues affecting how schools could spend the 

84 

money. 



Based on our survey sample of 106 schools, there were no statistically significant differences in academic growth 
on the CST between schools that carried over less then 25 percent and those that carried over 25 percent or more. 
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Re-visiting HPSGP Funding Leveis 

As we have already seen, the criteria used by the state to identify the award amounts delivered to 
program participants was designed so that each school would receive a uniform amount every 
year - $400 per pupil based on the 2000-01 enrollment of the school. If we take enrollment 
changes from year to year, as well as carry-over into account, however, the actual award amount 
per pupil available to HPSGP schools becomes more varied each successive year of the program. 
To understand why, consider a school with 100 students in 2000-01 that carries over 25 percent 
of its HP funds and loses 10 percent of its students each year. In Year I of the program, this 
school would have received $400 per student or $40,000 in total. If 25 percent is carried over, the 
amount available in Year 2 increases to $500 per student. If this school loses 10 students, the 
actual per student amount in Year 2 is $555 per student. This is an example of what happened 
over the course of the HPSGP implementation period for a number of HP Only schools. Not only 
did the intensity of the funds increase over time on average (due both to carry-over and an 
overall decline in enrollment of 6 percent between 2000-01 and 2005-06) but there was much 
more variation between schools in terms of the amount of HP funds per-pupil they actually had 
available to spend in a given year. Exhibit 5.10 illustrates how changes in enrollment and carry- 
over affected the actual per-pupil awards for HP Only schools. 

Exhibit 5.10. Total Award per Pupil HP Only Schools, 2002-03 to 2005-06 



Source: CPE HPSGP Expenditure Files 2002-03 through 2005-06, CBEDS SIF Files 2000-01 through 2005-06. 





Year 1 
2002-03 
(n = 338) 


Year 2 
2003-04 
(n = 349) 


Year 3 
2004-05 
(n = 349) 


Year 4 
2005-06 
(n = 347) 


(1) HPSGP Award 


a. Per Pupil Based en 2000-01 Enrollment 


$312 


$403 


$398 


$376 


b. Per Pupil Based en Year-Specific Enrollments 


$310 


$397 


$402 


$401 


(2) Total Available HPSGP Funds (Award plus Carry-over) 


a. Per Pupil Based en 2000-01 Enrollment 


$312 


$480 


$531 


$521 


b. Per Pupil Based en Year-Specific Enrollments 


$310 


$472 


$538 


$555 



Comparing rows (la) and (lb) of Exhibit 5.10 reveals the impact of actual enrollment changes on 
the per student award amount. Eor the first two years of the program (when total enrollment in 
HP Only schools exceeded the enrollment in 2000-01), per student funding amounts were 
actually lower than the amount stipulated by the program. In 2003-04 and 2004-05, however, 
enrollment in HP schools relative to 2000-01 declined by 1 and 6 percent respectively resulting 
in larger per student funding amounts. Comparing rows (la) and (2b) shows the combined 
impact on per student funding when both enrollment changes and carry-over are taken into 
account. After 2002-03, as carry-over funds were rolled into the yearly grant awards, actual per 
student funding amounts were substantially greater than the $400 per student stipulated by the 
program. In 2003-04, 2004-05 and 2005-06, the per-student dollars added due to carry-over (the 
difference between rows (2b) and (lb)) was $75, $136 and $154, respectively. 

In summary, HPSGP survey respondents reported that the funding received under this program 
was very instrumental to their school improvement efforts. Most also felt the amount of funding 
was sufficient to their needs. Only about half of the respondents reported the number of years to 
be sufficient in their opinion, and about one-third expressed concerns about the timeliness of the 
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arrival of these funds and confusion in regard to allowable carry-over. We further observe 
considerable variation in available funding, due to changes in enrollment and carry-over. 



Spending HPSGP Funds 

Spending on personnel was reported as the most common as well as the most 
effective use of HPSGP funds. 

School expenditures were guided in principle by the Action Plan: “Once the Action Plan is 
developed and implemented, expenditures must support the activities and strategies contained in 
the approved action plan and proposed budget.” These fairly open-ended spending guidelines 
may partly explain the relatively lower understanding of the uses of the HP funds among 
surveyed school respondents and varying degrees of district involvement with regards to 
overseeing expenditures. 

As shown in Exhibit 5.11 below, of the $542 mi llion awarded through 2005-06, schools spent 
$487 million in total - averaging $1.4 million per school and $1,361 per student. The remaining 
$55 million in unused funds (due to under spending - particularly in the first two years) was 
carried over into 2006-07 and had to be spent by the end of that school year. In fact, across all 
four years, 73 percent of the total available HPSGP funds (award plus any carry-over from prior 
years) were used - the remaining funds were carried over; a closer look at the reported reasons 
for carry-over follows later in this section. 

Exhibit 5.11. Total Money Spent by HP Only Schools, 2002-03 to 2005-06 



Source: CDE HPSGP Expenditure Files 2002-03 through 2005-06. 





2002-03 
(n = 338) 


2003 -04 
(n = 347) 


2004 -05 
(n = 348) 


2005-06 
(n = 347) 


Four Year 
Total 


Total Spent 
Average Per School 
Average Per Student 


$84,775,736 

$250,816 

$232 


$126,731,768 

$374,946 

$343 


$140,695,728 

$416,260 

$392 


$134,467,456 

$397,833 

$394 


$486,670,688 

$1,439,854 

$1,361 



* Year-specific enrollments for schools with non-missing values from the expenditure files were used to calculate these values. 



As part of the monitoring component, districts were required to submit year-end expenditure 
reports for each individual HP school, which specified the amount of program funds spent on the 
following categories: Certificated Salaries; Classified Salaries; Employee Benefits; Books and 
Supplies; Services and Operating; Capital Outlay; and Indirect Cost. Exhibit 5.12 details the 
average distribution of HPSGP expenditures across these categories per year among all HP Only 
schools. 

Across all years, investment in certificated salaries and books and supplies were the largest 
expenditure categories, each absorbing a little under a quarter of the available HPSGP funds. 
Spending on employee benefits and services and operational expenditures followed, each 
averaging about 7 percent of available funds. The smallest expenditures were on indirect rate 
(expenditures associated with administration, budgeting, payroll and other program management 
costs) and capital outlay (expenditures associated with acquiring land, purchasing buildings, 
improving school sites or expanding libraries). 
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Exhibit 5.12. Average Spending Profile among HP Only Schools (N = 351) 

Source: CDE HPSGP Expenditure Files 2002-03 through 2005-06. 
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29% 




24% 




18 % 




28% 




2003-04 2004-05 2005-06 

2 3 4 

Year 



27% 




3% 

2% 



24% 



7% 

4% 



25% 



ri Carried Over 
> Indirect Rate 
<5 Capital Outlay 

Services and Operational 

■ Books and Supplies 

■ Employee Benefits 

■ Classified Salaries 

■ Certificated Salaries 



Cumulative 



’Available Funds are defined as Yearly HPSGP Award Amount plus funds carried over from the previous year. 



Tracking spending over time reveals some interesting trends, namely the growing percentage of 
available funds spent on personnel (certificated salaries, classified salaries and employee 
benefits) and the diminishing pattern of expenditures on books and supplies over the first three 
years of the program. The underlying cause for this trend is likely associated with requirements 
built into state law (as a result of the Williams Settlement) mandating the state to ensure that 
students have up to date instructional materials that are aligned with State Board approved 
academic and content standards. 

The HPSGP application required schools that had not “already adopted or purchased appropriate 
SBE adopted or aligned materials” to do so “immediately” using HPSGP funds. The relatively 
larger initial investment in books and supplies indicates schools potentially prioritizing 
compliance with the law in Year 1. Once this requirement was met, it appears funds were freed 
for other priorities such as spending on personnel-related activities. In Year 4 (2005-06), 
spending on certificated salaries dropped by 9 percentage points and spending on books and 
supplies increased by 10 percentage points. Evidence suggests that this decline may be 
associated with the interplay between uncertainty, late arrival, and carry-over policy related to 
Year 4 funds (these circumstances will be discussed in more detail in the following section). 

Unfortunately, the labels used by the state to categorize HPSGP expenditures do not shed much 
light on the exact nature of the spending. Eor instance, we do not know if salary-related 
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expenditures went to increase staffing levels with new personnel or to invest in existing staff 
(e.g., supporting more instructional or professional development time). While the expenditure 
reports can be used to observe broad spending patterns across all HP schools, the school survey 
results provide a more detailed description of spending for a subset of schools. 



Exhibit 5.13 displays findings for the largest and most effective expenditure categories, as 
perceived by respondents to the school survey. This information was obtained from two school 
survey questions: one asked school respondents to select the three largest HPSGP expenditures 
from a list of categories (shown in the exhibit); the second requested schools to select the three 
most effective expenditures from the same list.*^ 



Exhibit 5.13. Perceptions of the Largest and Most Effective Uses of HPSGP Funds 
(N = 105) 

Source: HPSGP School Surveys (AIR) 



Expenditure Categories 



Supplemental Personnel 
Information Technology 
Staff Providing Additional Instructional Time 
Teacher Release Time 
Other* 

Intervention Program Materials 
Textbooks/ Instructional Materials 
Comprehensive School Reform Model Provider 

External Provider 
Additional Support Personnel 
Student motivation incentives 
Assessment/ Data Packages 
Parent Involvement Activities 




•Other includes combinations of two or more expenditure categories that were mentioned by respondents. 



Spending on supplemental personnel, reported by 47 percent of all respondents, was by far the 
most commonly cited primary use of the HPSGP funds. This category encompassed spending 
associated with instructional or administrative personnel such as teachers, literacy coaches or 
vice-principals. A distant second, information technology (such as computers, Internet access. 



For both questions, schools were allowed to write-in additional categories. 
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and software) was cited top expenditure for 12 percent of the schools, followed by additional 
instructional time (for example, paying teachers to stay before or after school) and teacher 
release time (for grade level meetings or other related professional development costs). Among 
expenditures reported as being the second or third largest (not shown in Exhibit 5.2), additional 
instructional time and teacher release time were most common. 

As seen earlier in Exhibit 5.7, nearly all schools had a positive perception of the effective use of 
HPSGP funds. In regards to the perceived effectiveness of individual expenditures, there is 
similarity with categories identified as the largest expenditure (the ranking by expenditure size in 
noted in parentheses): 43 percent of respondents cited supplemental personnel (1) as the most 
effective; 19 percent cited teacher release time (4); 12 percent cited additional instructional time 
(3); and 6 percent cited information technology (2). 

While the top four largest expenditures and the top four most effective expenditures coincide, 
they did not always match within the same school. More than half of the respondents selected the 
same category as both the largest and most effective funding use. Among these respondents, 61 
percent selected supplemental personnel as both the largest and most effective expenditure 
category. The disparity between the largest and the most effective categories - 46 percent 
selected different items for each category - suggests that some schools may benefit from 
additional guidance in effectively targeting their funds, as raised in the Action Plan section of 
this chapter. When controlling for student and school characteristics, there were no statistical 
differences between schools that reported personnel as their largest or most effective HP 
expenditure and other schools. 



Issues Affecting HPSGP Implementation 

Nearly 60 percent of surveyed school respondents cited major factors limiting their 
ability to use HPSGP funds to improve student achievement at their school. 

In order to better understand the relationship between spending HPSGP funds and student 
outcomes, school respondents were asked to describe through an open-ended question, what, if 
any, factors limited their ability to use these funds to improve student achievement. Exhibit 5.14 
displays several common themes that were categorized from their narrative responses. As 
responses could have encompassed several of these themes, the percentages in the table represent 
the number of times that an issue was mentioned and therefore the sum of the percentages 
exceeds 100 percent. About 17 percent of school respondents did not provide a response, and 
while another quarter either reported that they did not encounter any limiting factors or that the 
question was not applicable. When barriers were identified, however, the most frequently cited 
factors pertained to the lack of flexibility and district control (cited by 21 percent of 
respondents), delivery and clarity of funding (15 percent), and personnel challenges (15 percent). 
The following sections explore each of these factors in more detail, providing important context 



^^Schools that reported personnel as their largest HPSGP expenditure showed average annual growth of 0.03 
standard deviations in CSX math and 0.06 in CSX ELA, in relation to 0.09 and 0.04 standard deviations in math and 
ELA, respectively, in schools that did not report this. Schools that reported personnel as their most effective use of 
HP funds saw an average growth of 0.12 standard deviations in CSX math and 0.09 in CSX ELA, in relation to 0.02 
in both subjects for schools that reported other categories. 
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as to why schools were not able to harness the HPSGP funds as productively as they might have 
liked. 

Exhibit 5.14. School Respondent Perceptions on the Factors Limiting Use of 
HPSGP Funds 

Source: HPSGP School Surveys (AIR) 



What major factors, if any, limited your ability to use HPSGP funds to improve student achievement at 
your school? 



Themes 


N 


% 


No response 


18 


17% 


"None" or N/A 


28 


26% 


Lack of Flexibility, District Control & Bureaucracy 


22 


21% 


Funding Irregularities 


16 


15% 


(e.g., delays or uncertainty in funding; lack of clarity around carry-over 
policies) 


Staff Attrition & Hiring Constraints 


16 


15% 


(e.g., school administrative changes; lack of personnel and turnover) 


Unions 


4 


4% 


Need more time for school reform 


3 


3% 


Challenging student population 


3 


3% 


Other 


12 


11% 


Total School Respondents 


106 


- 



Lack of Flexibility, District Control and Bureaucracy 

The most commonly cited theme reported as limiting schools’ ability to use HPSGP funds to 
improve student achievement were bureaucratic controls, often stemming from state or district 
policies. Included in this theme were three distinct factors that were often interrelated: lack of 
flexibility (mentioned by 10 percent of respondents), strained district relations (8 percent) and 
bureaucracy (7 percent). The responses characterizing the funds as inflexible suggested that these 
schools had minimal control over how funds could be used. Reasons included restrictive district 
oversight, procedural hurdles associated with modifying the HPSGP Action Plan, and funding 
provisions associated with the timeline for spending and carry-over. 

Some respondents offered no explanations, simply stating that the funds were too restrictive or 
that certain types of spending did not qualify under the program. Six school respondents 
identified the district as a limitation, expressing frustration stemming from bureaucratic barriers 
such as slow approvals of expenditure requests, inadequate and untimely budget adjustments, 
and inconsistent and inflexible decisions made by the district around how funds could be spent. 
One reported that the district control “greatly impedes progress and ability for sites to spend their 
funding allocations.” Another respondent noted change in district personnel had resulted in 
“uneven and inconsistent policy” that impacted implementation. District irregularity was also 
observed by another school respondent who contended that the district initiatives conflicted with 
the Action Plan, “The district continues to push too many initiatives at once. Often these 
initiatives are not supportive of the HPSGP Action Plan. As a result, implementation of plans 
and strategies create more fragmentation and less cohesion.” 

A district that was in financial disarray and nearing state take-over was described by one 
respondent as a major obstacle, directly impacting improvement efforts under HPSGP: 
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The unrest and turmoil caused by cash flow prevented purchases from being 
made in a timely manner and more importantly was the cause for several of the 
better teachers deciding to leave the district. Unfortunately due to the dysfunction 
of newly elected school board members our district will once again lose good 
teachers, and we will be faced with the lack of leadership and support from 
district level personnel. . . .Unfortunately with the focus on our financial 
bankruptcy, there has not been a focus on our academic ‘bankruptcy.’ 

Funding Irregularities 

Funding issues such as delays or uncertainty in funding (14 percent of the respondents) and lack 
of clarity (5 percent) around carry-over policies were cited by school respondents as 
implementation factors limiting the effective use of HPSGP funds. 

Carry-Over Concerns 

Case studies conducted in Year 1 of the study illuminate some of the reasons schools did not 
spend the full amount of the grant. Among the more salient reasons were changes in school 
leadership, late notification or unclear communication from the state regarding the delivery of 
funds and carry-over policies (especially with fourth year funds), lack of oversight or guidance in 
relation to decision making around spending, and strained relations with districts due to financial 
or governance crises. When these factors coexisted, they created conditions that made it very 
difficult for schools to make timely and thoughtful decisions on how to spend the grant money, 
often resulting in ineffective investments and significant carry-over. 

Through our school surveys conducted in Year 2, we asked schools with carry-over exceeding 25 
percent to identify the causes of the carry-over. Exhibit 5.15 shows that among our respondents, 
33 percent indicated that they did not have carry-over exceeding this amount (this is supported 
by the HPSGP expenditure data which show that these schools carried over on average about 17 
percent of their available funds annually). The remaining two-thirds of respondents did report 
carry-over exceeding 25 percent (average annual carry-over among these schools, according to 
the CDE data, was 30 percent). 
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Exhibit 5.15. School Respondent Reported Reasons for Carry-Over 

Source: HPSGP School Surveys (AIR) 

If your school had substantial carry-over (e.g. more than 25 percent) of HPSGP funds in a given year, what 
were the reasons for this?* 



Categories N % 



No substantial carry-over of funds 


35 


33% 


At least 25 percent carry-over 


69 


65% 


Missing 


2 


2% 


Reasons for carry-over** 

Late notification of funds 


23 


33.3% 


Late receipt of funds 


16 


23.2% 


Used other available funding first 


24 


34.8% 


Changes In school leadership 


23 


33.3% 


Other 


12 


1 7.4% 



*This question specifically asked respondents to exclude 2005-06 due to funding being released at the start of the 2005-06 school 
year. 



** Reasons analyzed only for schools that indicated substantial carry-over. Percentages add up to more than 100% as respondents 
provided more than one answer. 



Among the schools reporting 25 percent or more in carry-over funds, all four of the reasons cited 
as causing them were generally identified as pertinent. Reasons related to implementation 
concerns such as late notification, late receipt, and changes in school leadership were nearly as 
commonly cited as the more strategic response of using “other available funding first.” 

Late notification and late receipt of funds imply funding irregularities: either schools were not 
aware they would be receiving funds or they received funds so late they were not able to fully 
mobili z e the money before the year’s end. Responses from open-ended questions among school 
and district respondents reveal additional irregularities and uncertainty regarding carry-over 
policy. Eleven percent of the district administrators who were surveyed reported that not 
knowing when funds could and could not be carried over was a source of confusion and 
frustration. 

Responses in the open-ended “other” category in this survey item revealed schools making 
strategic choices to defer spending in order to assess programs before investing in them, to 
realign the single site plan to reflect changing needs and in one case to “extend the benefits of the 
grant into future years where funding would be less.” One respondent noted that insufficient 
funds in the district created a “cash-flow” problem that limited the school’s ability to make 
purchases in a timely manner, and another mentioned that the inability to find a provider for AB 
466 teacher training as the cause for delayed spending. 

Late Funding Concerns 

In the summer and fall of 2003, 98 additional schools were selected for participation in HPSGP 
(of which 79 were HP Only Schools) when additional state funds became available. Exhibit 5.16 
displays the funding timeline for Year 1 implementation funds. Schools funded in the first round 
received the award in November 2002 and March 2003. There were two groups of schools that 
received the second round of funding: schools that were notified of the award in May 2003 and 
received the funds in June 2003, and those that were notified in November 2003 and received the 
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award in December 2003. When schools received the funds, not only did they receive almost 
two years worth of funding but they were also beginning Year 2 of the program. Furthermore, 
the one-month planning window between the notification and release of funds for the second 
round schools was much shorter than six months available to schools part of the first round of 
funding. 



Exhibit 5.16. Implementation Funding Timeline for First and Second Round 
Funding 

Source: Communication with School Improvement Division, CDE 



1 

N 



Schools Receiving First Round Funding 
Schools Receiving iSecond Round Funding 
Receipt of FIPSGP Funding Money 
Notification of Award 



0 s 



$ 
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The CDE expenditure reports show that between the 2003-04 and 2005-06 school years, schools 
funded in the second round consistently carried over a higher percentage of their available funds 
(33 percent) compared to schools funded in the first round (24 percent). One respondent from a 
second round school reveals the challenges they faced in using HP funds: 

...The uncertainty of funding [and] receiving the funds late negatively affected 
our ability to truly make thoughtful decisions that would result in increased 
student learning. We did not receive our Year 1 funding until the beginning of 
Year 2. In one year, we had two years worth of funding. We were informed that 
carry-over would not be allowed. Hence, it precipitated a series of decisions that 
were not as sound as they should have been had we had accurate and timely 
information. 

Another school respondent who received the funds in May 2003, “We felt behind the whole four 
years of the grant.” 

Funding Delays and Uncertainty 

As referenced in the above quote, the combination of receiving late funds and the uncertainty of 
future funding disrupted the school’s ability to plan strategically. Responses to the school survey 
suggest that funding delays were not exclusive to schools funded in the second round in the first 
year of implementation and in the fall of the second year. As shown earlier in Exhibit 5.14, 15 
respondents (17 percent) indicated that funding delays had adversely affected how HP funds 



These schools were identified for funding late due to the legislature releasing additional funds through 
supplemental budget bills. 



American Institutes fer Research 



Page 1 23 








Evaluation of the High Priority Schools Grant Program 



were used. Only three of these were among the group of schools funded in the second round. 

One respondent mentions how a clear plan for improvement is limited by an unclear funding 
schedule: 

Our school had visions of creating a group of teachers that were highly .skilled 
and research driven. We wanted to create a team of educators that used data to 
drive instruction and had time to review data, plan instruction, meet 
collaboratively and plan as grade levels. To some degree these goals were met 
through HPSGP. However, when funding was delayed and adequate time was not 
available, we were not able to fully implement our plan as we intended it. 

Both district and school respondents raised concerns that highlight how uncertainty in funding 
procedures and timelines, coupled in some cases with late notification and arrival of funds, 
undermined both their planning efforts and their ability to make decisions that aligned with 
established priorities. The 15 respondents who noted funding delay challenges were fairly evenly 
split between traditional calendar and year-round schools. Four school respondents said that the 
arrival of funds was out of step with the academic cycle in year-round schools: “Receiving 
funding in September or October puts a hold on how we spend the money effectively to provide 
our students academic success when our school year actually begins in July.” In two other school 
sites and one district, respondents described how the arrival of funds late in the school year was 
not aligned well with the hiring cycle, thus creating difficulty for schools wanting to hire 
qualified personnel for needed intervention or support positions such as literacy and math 
coaches. 

Fourth Year Funding 

As observed during our case studies and further supported by the survey responses, the 
provisions associated with the fourth year funds were particularly problematic. The way the law 
is currently written, it was not possible to know if sites would be eligible to receive fourth year 
funding until after the third year test scores were received and fully analyzed by the state. As 
eligibility for fourth year funds was just determined the summer before, several case study and 
survey schools - not knowing they would have additional funds available - released key 
personnel paid for by HPSGP. The combination of the late arrival of funds and the initial 
prohibition of carry-over in Year 4 created substantial pressure among schools and districts to 
spend the available funds before the deadlines (including internal district deadlines well in 

oo 

advance of the state’s original June 2006 deadline). This oftentimes caused schools to divert 
funds away from established priorities towards rushed, one-time purchases, and potentially 
explains the spike in spending on books and supplies and a decline of spending on personnel in 
Year 4 compared to previous years (see Exhibit 5.12). 

Although the fourth year funding may be considered an anomaly, responses to the surveys 
suggest that spending headaches continued beyond the fourth year. One district administrator 
revealed a situation at the end of the 2006-07 school year (after HPSGP had ended) where 
spending residual HPSGP funds was rushed and not necessarily aligned with any plan: “Between 
now (April) and June 30, we’re spending close to $30 million of carry-over funds. Schools are 
unloading the money, and all they can spend on now is on products and goods. . ..” 



CDE in May 2006 allowed funds to be carried over into 2005-06. 
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These experiences underscore the importance of consistency and timeliness in the delivery of 
funds. Irregularities in funding contributed at times to ineffective decisions and 
counterproductive spending patterns. If there is uncertainty in funding, a school’s ability to make 
decisions, plan, and make financial commitments is very limited. For example, if a school wants 
to buy books or computers, it cannot place orders; if a school wants to hire someone, it cannot 
commit to hiring. The funding irregularities, where experienced, seem to have undermined 
schools’ ability to engage in a committed and cohesive spending plan. 

Staff Attrition and Hiring Constraints 

Fifteen percent of the school responses described difficulties associated with personnel - 
primarily resulting from turnover at both the administrative and teacher level and hiring 
constraints. Turnover at the principal level drew concerns primarily around the consistency of 
vision, focus, and coherence of reforms initiated under HPSGP. As one school respondent noted, 
“Turnover in leadership disrupted the coherence of the reforms as each new principal had to get 
up to speed on the plan and the tenets of HPSGP in addition to becoming familiar with his/her 
new school and staff.” It was also noted as a reason for carry-over in one-third of the schools that 
reported substantial carry-over. 

Teacher attrition caused challenges for similar reasons. One school respondent wrote that 
disruptions in personnel resulted in “uneven and inconsistent policy.” Another respondent 
described how turnover diluted the impact of professional development: “Forty percent of the 
training the school received was lost due to high turnover,” while another described how attrition 
offset the momentum of reform: “The cycle of new teacher training was continuous because we 
had to constantly train new staff. Reform is hard to maintain at a high level when staff is new and 
need to be trained. It prevents the whole school from moving at the pace it needed to.” 

Difficulties hiring personnel to meet key staffing needs were also mentioned; one respondent 
cited a shortage of teachers to teach an extended day program and the reluctance of the human 
resources office to hire. Another reported that shortages in the supply of qualified math teachers 
posed hiring challenges, and in one case the short-term nature of the HP funds was cited as 
limiting the schools’ ability to make the long-term financial commitment. Loss of experienced 
and qualified staff due to frustrations with school and district administration was also identified 
as a factor that limited the use of program funding. 

Other Factors Limiting the Use of HPSGP Funds 

Some of the other limiting factors cited less often relative to those just discussed included 
teacher union contracts, limited time available for reform, and the challenges associated with 
teaching high-need student populations. For example, four school respondents felt that the 
structure of teacher union contracts was “constraining” and “limiting” in regards to using the 
funds. When time was cited as an issue, schools generally felt that more time (more specifically, 
a longer funding cycle) was needed not only to achieve their reform goals but also to sustain 
them. 

Other isolated factors mentioned by respondents included confusion about state policies in 
regards to allowed expenditures, misalignment of the HPSGP funding cycle with the NCLB 
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Program Improvement cycle, investments in staff (professional development and collaboration) 
that did not yield the desired changes in instruction, and a weak action plan guiding spending. 



Summary 

As noted in Chapter 3, despite the considerable resources invested in this program, no 
corresponding boost in test score performance was discerned among HPSGP schools in relation 
to similar schools not participating in the program. This chapter explored issues regarding some 
of the key elements associated with the HPSGP implementation that may inform those results. 
How salient was the required HPSGP Action Plan perceived to be among program participants? 
To what extent was the required external provider component perceived as useful? To what 
extent were districts perceived as supportive in the implementation of the program? 

Perceptions were mixed in all of these inter-related components of the program. The major 
rationale for the external provider component of the HPSGP was to assist schools to thoughtfully 
assess their strengths and weaknesses and to jointly formulate an Action Plan for reform. While 
47 percent of school respondents found this support to be quite helpful, nearly half reported not 
using an external provider, or did not know if the school utilized one. As this was a required 
program component, the fact that a positive impression (or any impression) was so often missing 
seems a major breakdown in regard to HPSGP implementation. 

Given that HPSGP participants are by design among the state’s most struggling schools, some 
form of special and supplemental district support and attention to allow them to substantially turn 
around in a relative short four year period would seem an essential component of success. While 
the majority of school respondents indicated that the district had actively supported them during 
the implementation of the program, only about 13 percent of respondents “strongly agreed” that 
this had been the case, and a third disagreed with this point. This seems another important area of 
implementation breakdown for a substantial proportion of participating schools. 

In addition to the important implementation components listed above, HPSGP is above all else 
primarily a grant program as is reflected in its name, whereby participating schools receive an 
infusion of funds. Given this, the areas of funding and spending would seem crucial to effective 
program implementation. To what extent did the funds reach the schools in a sufficiently timely 
fashion to allow thoughtful spending? Were the rules governing these allocations in regard to 
how they could be spent well understood? Did participating sites encounter limitations in using 
HPSGP funds to improve student achievement? 

Again, the results for funding and spending were mixed. Virtually all sites found the money 
critical to their school reform efforts and that the amount of funds was sufficient to address their 
reform challenges. However, only about half of the responding sites felt that the time period for 
reform was sufficient, and one-third expressed concerns about the timely receipt of funds and 
carry-over provisions. While more than 40 of the school respondents did not provide a response 
or reported that there were no major factors limiting their ability to use HPSGP funds to improve 
student achievement at their school, nearly 60 percent raised concerns such as lack of flexibility, 
carry-over, funding timeliness, and staff turnover. 
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As a whole, there appears to be considerable implementation break-downs that may help explain 
the limited program effect observed in the aggregate on student achievement, and may also point 
to areas for improvement in future policy formulations. 
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Chapter 6: Perspectives on the Impact of 
HPSGP 



Introduction 

While Chapter 3 reported little to no HPSGP effeet on student aehievement, the majority of 
sehool survey respondents were more positive in their subjeetive determinations of the program’s 
impact on their schools. However, while the results shown in this chapter indicate fairly positive 
perceptions from school leaders participating in this program, we are not able to compare these 
results with similar, non-participating schools. For example, it may be that the small majority of 
school respondents citing 
lasting HPSGP impact in areas 
of school capacity may not 
differ from the perceptions of 
bolstered school capacity in 
schools not in the HPSGP. 

Nonetheless, it is interesting to 
note the perceptions of 
participating school 
respondents regarding the 
perceived impact of this 
substantial state investment, as 
well as the specific areas of 
impact. This chapter is divided 
into three main sections. The 
first discusses the perceived 
importance of the HPSGP on 
the school’s improvement 
efforts. Next, we turn to the 
perceived impact of the 
program, both at the school 
level as well on the district 
support schools received. Last, 

this chapter reports on school and district respondents’ views of their school’s ability to sustain 
what they have been able to achieve during program implementation over time and the district 
role in sustaining reform. 



Key Findings 

• HPSGP was perceived as having a major role in 
student achievement gains. However, these positive 
perceptions should be tempered with the nearly 
identical academic performance during this period 
between HPSGP and non-HPSGP comparison 
schools. 

• A slight majority of respondents reported a lasting 
HPSGP positive impact on school capacity. 

• While 60 percent of school respondents indicated 
confidence in sustaining the impact of HPSGP, only 
40 percent reported they had been able to find 
funding to continue these reforms. 

• Nearly 84 percent of school respondents expressing 
confidence about continuing reform characterized 
their districts as being highly or reasonably 
supportive, whereas only 63 percent of respondents 
from the less confident schools did so. 
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Perceptions of HPSGP Overall Importance to School 
Improvement 

HPSGP was perceived as having a major role in increasing student achievement 
among respondents who believed their school experienced substantial gains during 
the period of program implementation. These positive perceptions should be 
tempered, however, with the nearly identical academic performance during this 
period between HPSGP and non-HPSGP comparison schools. 

Both the school and district surveys measured the perceived overall importance of the program to 
school improvement. As shown in Exhibit 6.1, the vast majority of schools believed the HPSGP 
was very important to the school’s progress during the five years of program implementation. 
Across the school types, the program registered highest in high schools, with all agreeing that 
HPSGP was a significant factor (more than 60 percent strongly agreed). In contrast, some 
elementary and middle schools, as well as five districts (10 percent), disputed the overall 
significance of the program. Of these districts, one credited the district for the progress made, 
particularly since the state HPSGP guidance was perceived as lacking, and two others pointed to 
the schools’ failure to make gains as reasons why they believed the HPSGP had weaker 
importance. One school respondent who strongly disagreed that the HPSGP was critical to its 
reform attributed the school’s success to its Reading First program. In a similar fashion, over a 
quarter of the districts acknowledged that the Program Improvement, SAIT, and Reading First 
programs carried equal, if not greater, weight in school reform efforts. 



Exhibit 6.1 . Survey Responses to “The HPSGP has been very important to my 
school’s progress over the past five years” 

Source: HPSGP School and District Surveys (AIR) 
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Aside from the funds, which ranked high on districts’ reasons for why the HPSGP was perceived 
as instrumental, 20 percent of the districts believed the program was important as it provoked a 
meaningful dialogue about change or provided much needed focus and structure for tackling 
reform. As one district respondent described, “That was the [program] that got enough people 
involved and upset because things had to change. It stopped people in their tracks to have a real 
honest discussion.” The program requirements, including the needs assessment and external 
provider, provided the leverage to institute necessary change in schools with reluctant staff. 

While respondents generally believed that the HPSGP was very important to their progress over 
the past five years, as shown in Exhibit 6.1, the program seemed to have greater impact on some 
measures in relation to others. School respondents were asked to assess the role that the HPSGP 
played in the changes observed in regard to students, staff, and resources. For each of these 
areas, schools reported whether they experienced substantial or marginal gains, declines, or no 
change over the past five years, as well as the degree of impact of the HPSGP (e.g., major, 
moderate, minor, or no role). Exhibit 6.2 focuses on schools in which survey respondents 
reported substantial gains and the perceived impact of HPSGP on those gains. 

Exhibit 6.2. School Respondents’ Perceptions of the Role of the HPSGP in 
Schools that Reported Substantial Gains on Certain Outcomes 



Surveyed schools that 
reported substantial 
gains 



% of schoois with substantial gains that reported that 
the HPSGP had... 







N 


% 


Major Role 
in Change 


Moderate 
Role in 
Change 


Minor Role 
in Change 


No Role In 
Change 


a. 


Student achievement 


67 


63.2 


71.6 


25.4 


3.0 


- 


b. 


School administration 
knowledge/skills 


69 


65.1 


60.9 


26.1 


10.1 


2.9 


c. 


Teacher knowledge/skills 


72 


67.9 


69.4 


22.2 


8.3 


- 


d. 


Teacher collaboration 


68 


64.2 


76.5 


22.1 


1.5 


- 


e. 


Student 

attitude/motivation 


47 


44.3 


57.5 


29.8 


8.5 


4.3 


f. 


Student attendance 


27 


25.5 


55.6 


22.2 


14.8 


7.4 


g- 


Student discipline 


28 


26.4 


53.6 


25.0 


14.3 


7.1 


h. 


Parental involvement 


27 


25.5 


59.3 


37.0 


3.7 


- 


i. 


Community support 


25 


23.6 


56.0 


36.0 


8.0 


- 


j- 


Use of assessment data 
to plan instruction and 
curriculum 


70 


66.0 


74.3 


20.0 


5.7 


- 


k. 


Number of instructional 
staff 


31 


29.2 


83.9 


9.7 


6.5 


- 


1 . 


Level of non-personnel 
resources (e.g., 


58 


54.7 


81.3 


15.5 


3.5 


.. 



materials, supplies) 



One school did not provide a response regarding change for all outcomes. Non-personnel resources had two 
schools missing. 
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Exhibit 6.2 shows that about two-thirds of the surveyed school respondents reported substantial 
gains in several key measures over the past five years. Given the primary objective of the 
HPGSP, undoubtedly the most important of these is their perceptions regarding gains in student 
achievement; 70 percent of the schools with reported substantial gains in student achievement 
attributed a major role to HPSGP.^° These positive perceptions associated with the primary 
outcome goal of the program, i.e., gains in student achievement, are substantiated by the 
academic gains made on average by HP Only schools during this period. On the other hand, they 
are tempered by the nearly identical academic performance during this period between HPSGP 
and comparison schools when controlling for student and school characteristics as shown in 
Chapter 3. 

Other frequently cited outcomes with substantial gains included teacher knowledge and skills, 
the use of assessment data to plan instruction and curriculum, teacher collaboration, and school 
administration knowledge and skills. On the other end of the spectrum, only about a quarter of 
the schools reported substantial gains in community support, parental involvement, student 
attendance and discipline. Although only 29 percent of the surveyed schools cited substantial 
gains in the number of instructional staff, the perceived impact of the program was greatest for 
this domain, with 84 percent of respondents reporting the program had a major role in this gain. 
A similar pattern existed for the level of non-personnel resources, with a little more than half of 
the surveyed school reporting substantial gains in this area and 81 percent of those schools 
attributed the HPSGP as having a major role. On outcomes in which fewer schools reported 
gains - such as student attendance and parent/community support - the HPSGP was perceived as 
playing a lesser role. Schools with substantial gains were statistically significantly more likely 
than those with marginal gains to report that the HPSGP had a major role across all outcome 
measured. 



Perceptions of HPSGP Impact 

While it is encouraging that a slight majority of respondents indicated as lasting 
impact of HPSGP in areas ofschool capacity, this may also be seen as somewhat 
disappointing given the considerable investment in school capacity that HPSGP 
represents. 

This section on the perceived impact of HPSGP is divided into two main components. As part of 
the narrative section of the survey, school respondents were asked to identify the lasting impact 
of the HPSGP, if any, at the school level. This discussion is followed by perceptions of HPSGP 
impact on the amount of district support received by responding schools. 



When controlling for student characteristics, schools with reported substantial gains experienced an average 0.10 
standard deviation growth in CSX math and .08 in CSX ELA from 2001-02 to 2005-06. By contrast, schools that did 
not report substantial gains had an average 0.01 standard deviation growth in CSX math and -0.01 standard 
deviations in CSX ELA during that same period. 

Significant at the 1 percent confidence level. 
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Perception of HPSGP Impact at the School Level 

Among the surveyed schools, 13 did not answer the open-ended question about the lasting 
impact of the program. Many of the responses shown in Exhibit 6.3 overlap considerably with 
the HPSGP expenditure categories described in Chapter 5 of this report. Given these similarities, 
it was not always clear if the responses to the narrative question truly reflected ongoing, lasting 
practices as opposed to describing how the HPSGP funds were used during the grant period. 
Nonetheless, the responses provide additional understanding of how these funds were used 
strategically to enhance school capacity. The responses were coded into four major areas: school 
capacity, resources, instructional programs, and school culture. Trends in reporting within each 
of these four major sub-areas of perceived impact as well as selected quotes underlying these 
responses are reported below. 

Exhibit 6.3. School Respondents’ Perceptions of the Lasting Impact of the HPSGP 



Source: HPSGP School Surveys (AIR) 



To date, what has been the lasting impact of the HPSGP in your school, if any? <1> 




N 




% 




No response 


13 




12% 




Schooi capacity* 


60 




57% 




Professional development 




41 




39% 


Staff collaboration 




27 




25%, 


Use of data to inform instruction 




25 




24%, 


Resources* 


40 




38% 




Additional Staff 




19 




18%, 


Technology 




18 




17% 


Materials 




21 




20% 


Instructional programs* 


34 




32% 




Changes in instruction/delivery 




13 




12%, 


Enhanced programs (e.g., interventions) 




24 




23%, 


School culture / Focus on achievement 


21 




20% 




Parent outreach 


8 




8% 




Other 


13 




12% 




Total School Respondents 


106 




- 





<1> This was an open-ended question in the school survey. Responses were coded into broad themes. 

Note: As respondents provided multiple examples of the impact of the program, the sum of all categories will add to more than 100 
percent. Percentages of the main categories are based on the entire survey sample of 106. 

* The total for these categories is an unduplicated count and will not add up to the sum of the sub-categories, as school respondents 
reported multiple answers. 

School Capacity 

The most commonly reported perception of lasting impact was in school capacity (57 percent) in 
such areas as enhanced professional development (both in quality and quantity), staff 
collaboration, and the use of data to inform instruction. These findings suggest that more than 
half of the surveyed schools, in some form or another, believed that the program had maintained 
lasting progress in capacity building areas such as enhancing teacher knowledge, opportunities 
for teachers to collaboratively plan classes, and fostering a teaching culture based on evidence. 

For example, one school respondent reported, “The HPSGP process continues to help us create a 
shared vision through team effort. We establish and maintain a coherent, ongoing professional 
development program in which we focus on analyzing student work and writing aligned to state 
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standards.” In addition, among schools that reported at least one of these school capacity 
categories, nearly half reported at least two of these areas working in tandem. For example, staff 
development generally revolved around data analyses, standards and content knowledge, and 
effective instructional strategies (including differentiated instruction). 

Through the HPSGP, respondents at a quarter of the schools reported more formal opportunities 
for collaboration; one school instituted weekly planning meetings, while another moved to a 
block schedule to provide for more collaboration time. Other schools took advantage of the 
release time to use teacher observations to build “better skills by seeing good instructional 
practices modeled on a regular basis.” Extending beyond strictly the instructional staff, one 
school noted multiple levels of collaboration: principal to principal, principal to district 
personnel, and teacher to teacher. 

Nearly a quarter of the responding schools cited data use as particularly helpful for teachers to 
identify and appropriately target students for interventions, as well as grounding other decisions. 
As one school wrote, “Decisions about what we teach students and how we allocate our 
resources is more strongly tied to relevant data. It’s about student achievement and not what the 
teacher has ‘taught.’” 

Resources 

Nearly 40 percent of responding schools cited greater levels of resources as a lasting impact of 
the program, namely supplemental staff, technology, and materials and supplies. Again, while 
this may be seen as a positive result by some, it also raises questions as to why resources were 
not cited more often in relation to a program allocating such substantial fiscal resources to 
participating schools. 

The finding that less than 20 percent of the surveyed schools identified additional staff as a 
lasting impact seems to align with the resource analysis shown in Chapter 4 of this report 
showing little difference in staffing between HP Only and other schools well into program 
implementation. Nonetheless, some schools did report lasting personnel impact, such as class 
size reduction. One school described the benefits of additional instructional personnel for this 
purpose as allowing for more personalized instruction and eliminating combination classes. 
Supplemental resource personnel, such as literacy coaches, teachers on special assignment, and 
technology personnel, were also identified as ongoing supports. Another school invested 
considerably in personnel, which included long-term substitutes to work alongside teachers to 
implement interventions, teachers for before- and after-school programs, a literacy coach, a 
technology coordinator, and noon directors. While survey respondents tended to report such staff 
investments as among the most effective (Chapter 5), they also may be often unsustainable once 
the grant ends. 

Some of the other reported areas such as improved student and teacher access to technology may 
have a more sustained resource impact. Acquisitions included software, computer labs (including 
mobile labs), and printers, which according to one school, “brings classrooms into the 21st 
century.” Although described in less detail, 23 percent of the school respondents referred to 
instructional materials, such as intervention packages such as High Point, and textbooks. 
Although these items may be more easily sustained than on-going expenditures, school 
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respondents perceived them to be less effective in relation to personnel investments (e.g., 
supplemental staff, professional development), as shown in Chapter 5. 

Instructional-Related Changes 

Nearly a third of the examples cited as long term impacts of HPSGP fell under the broad 
category of instructional-related changes. During the course of the program, schools adopted 
standards-based instructional strategies and implemented unified curriculum in an effort to raise 
student achievement. Almost twice as many schools, however, cited bolstered instructional 
interventions, primarily tutoring, extended day programs, and Saturday school. Some 
interventions had a specific focus, such as CAHSEE or Academics Skills Classes. However, one 
school noted the challenge of continuing this support in the near future, again raising questions 
about lasting impact: “We have a system that works: Before school, after school, Saturday 
School and off track tutoring. However, since this program is very expensive, it will be difficult 
to maintain over the next three years.” 

Positive School Culture 

About 20 percent of the schools reported a heightened focus on student achievement, clearer 
vision, and improved school atmosphere as a result of higher expectations. Some of these 
changes were attributed to other changes noted above, such as professional development and 
increasing use of data. 

Parent Outreach 

Eight schools cited increased parent involvement through parent centers and education classes as 
having a lasting impact and in the words of one school, “a direct and tangible impact on student 
learning.” Two schools had fully staffed centers, and others noted improved student-parent 
communication through newsletters and autodialer. One school used the HPSGP to fund a 
student intervention specialist to serve as a liaison between the school and the community. The 
specialist worked with students on a personal basis and makes phone calls and home visits on a 
regular basis. According to the school, as a result there had been an improvement in student 
attendance, student behavior, and student academic performance. 

Other Examples 

School respondents described other lasting impacts of HPSGP. Some highlighted improved 
outcomes, such as meeting the federal AYP, recently exiting Program Improvement status, and 
higher rates of college-bound students and students taking rigorous coursework. Other outcomes 
included stable student attendance and increased teacher retention. Two schools referenced 
ongoing assessment of reform efforts - one through the action plan and the identification of 
schoolwide goals. Another continued to utilize the external provider to help implement and 
revisit strategies. 

Perceptions of HPSGP Impact on District Support 

While HPSGP figured fairly prominently in school-level reform, the perceived impact of HPSGP 
on the district’s role in supporting the schools was mixed. As discussed in Chapter 5, the 
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Q9 

program specified a greater role for the districts in relation to its predecessor, the II/USP. Both 
the school and district surveys included a question on the degree to which the district was more 
active in supporting the schools as a result of the HPSGP. As shown in Exhibit 6.4, compared to 
their perceptions of the program impact on their own reform efforts, school respondents more 
frequently disagreed that the HPSGP had an impact on districts. With 70 percent of district 
respondents strongly agreeing or agreeing, they were more confident than school respondents 
that the HPSGP influenced how the district supported low performing schools. Districts that 
described themselves as being focused on school improvement and having cohesive goals were 
statistically significantly more likely to perceive an impact of the HPSGP on the district, than 
those that were in the process of developing their goals. At the school level, high schools and 
middle schools were fairly similar, with around 60 percent observing more active district support 
due to the HPSGP. By contrast, more than half of the elementary schools did not believe the 
program had an impact on district support. 



Exhibit 6.4. Survey Responses to “I believe that the central district has taken a 
more active role in supporting my school as a result of the HPSGP” 



Source: HPSGP School and District Surveys (AIR) 
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Note: Percentages will not add up to 100 percent due to missing responses. Charter school responses were re-coded as “missing.” 



The guidelines for the first cohort of HPSGP schools noted, “The role of the district is greater under HPSGP than 
under II/USP.” 

When limiting the sample to the 86 schools for which we obtained both school and district surveys, districts were 
still more likely to report a program impact (69 percent), in relation to schools (56 percent). 

Districts were grouped according to their responses to a survey question which asked respondents to select one of the 
following as hest describing their district: a) The district has deal", cohesive goals and is focused on supporting school 
improvement; h) The district is developing or reforming school improvement goals and support; or c) The district does not have 
clear school improvement goals and support. 

Significant at the 5 percent confidence level. 
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Across all school types, the perception of impact of increased district support for HPSGP schools 
as a result of this program was mixed. A majority of schools (54 percent), however, did attribute 
change in the district role as a result of HPSGP. Some may see this as a positive result, while 
others may view this as a reason for concern given the more clearly stated role of the district 
through this program. However, the data above may understate the degree of district support for 
these schools in that at least half of the 14 district respondents noting little impact of HPSGP on 
its support for participating schools (as shown in the last set of columns above) clarified that they 
were supportive, but not necessarily as a consequence of the program. Of those, one indicated 
that the district support was fairly recent, while five others attributed their heightened district 
involvement to the SAIT or Program Improvement programs. 

Districts that did report a program impact on their role were asked to describe the influence of 
HPSGP. Although several referred to the additional funding that supported school-level reform, 
more respondents cited the program focus and the needs assessment for improving their district’s 
understanding about the schools to target and the fundamental changes and resources necessary 
for reform in those schools. The process promoted a dialogue about change and illuminated 
needs that some districts were not aware of previously. The program structure allowed districts 
to better target and prioritize their existing support, such as professional development. As one 
respondent described, “Before [HPSGP], we were trying to figure out what the priorities have to 
be. It allowed us to formulate a framework for supporting low performing schools with very 
specific goals. We prioritized the schools so the way we provide support changed. We don’t just 
provide equal support all over the place.” 

One district acknowledged the particular advantage of joining forces with the HPSGP schools to 
maximize the results: 

Because they had the HP funds as well as the other supports in the program, it is 
a win-win for us to partner with them because we know we can make a 
difference. When we bring what we have to the table and the schools in HP bring 
what they have to the table, collectively we can make a difference in moving 
forward. We can really make sure that it happens. 

Another district respondent said, “It has made us more aware that we need to be active partners, 
and that in order to secure and ensure school success, the district office needs to be there in the 
trenches.” 

In some instances, the HPSGP was said to indirectly affect non-participating schools as well. 
Greater awareness of school needs and more effective support (e.g., professional development, 
monitoring) was seen as benefiting all schools, and some districts reported implementing district- 
wide reforms which replicated some of the efforts initiated under the HPSGP. As one noted, “We 
did not just let the other schools not have support.” Interestingly, it compelled one district to 
level the playing field for the non-participating schools: 

Since we were keenly aware of the funds the HP school had, we wanted to 
maintain equity. We strived harder in trying to find creative ways to provide 



American Institutes fer Research 



Page 1 37 




Evaluation of the High Priority Schools Grant Program 



funding for the other schools. We want all of our schools to feel as if they are 
‘have’ schools rather than ‘have-nots.’ I think that was an important motivator 
for us to look for funding for other schools. 

The statement of above is worthy of note, and could be interpreted in several ways. For example, 
to the extent that enhanced training designed to meet the needs of HPSGP schools can be 
leveraged to benefit other schools as well with no corresponding diminished benefit to the 
HPSGP schools, this kind of spillover effect would seem beneficial. It might also at least 
partially explain how non-HPSGP schools could have benefited from the program such that no 
substantial test performance differential between HPSGP and comparison schools is seen. 

On the other hand, the statement above can be seen as a potential cause for concern in regard to 
program implementation. To the extent to which HPSGP funds going to targeted schools are 
causing a disproportionate allocation of other district funds to non-HPSGP schools in an attempt 
to “maintain equity,” it is possible that HPSGP funds are simply displacing other funds these 
schools would have received through the district. Such a practice would undermine a major 
objective of the HPSGP, which is to provide the state’s most struggling schools with a resource 
supplement in relation to what they would have received anyway. 

School respondents indicating that the program did not have a major effect on shaping or 
improving district strategies and attitudes towards participating schools cited the lack of specific 
guidance from the state and the impact of heightened district- level accountability from other 
state and federal programs. In explaining the reasons why there was little effect of the program 
on the district’s support of HPSGP schools, three district respondents compared the HPSGP to 
the highly structured SAIT: 

“There was really no impact [on the district] until we had the schools starting to 
fall into the SAIT process. That brought attention with things that need to occur 
there as well. The district looked at the HPSGP as something that the schools were 
responsible for implementing and maintaining.” 

“The HPSGP guidance from the state was blurry; we just treated them like other 
schools in the district. . . . [District support] was more related to the sanctions of the 
SAIT process and the PI process. It forced us to look at how we provide support.” 

“There are no teeth behind the HPSGP as opposed to when we look at the SAIT 
schools - there's a difference in those schools. I can see a difference because of the 
local district participation and the mandate that they are part of the district-school 
leadership team piece, where none of that was required in the HP schools.” 
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Sustaining the Impact of HPSGP 

While 60 percent of school respondents indicated confidence in their ability to fully 
continue their school improvement efforts even though the HPSGP is over, only 40 
percent reported they had been able to replace HPSGP funds, or re-allocate current 
funds, to continue the reforms initiated under the program. 

Considering the investment of more than $540 million in implementation funds, one major 
policy concern is the capacity of schools to sustain student achievement growth and to continue 
other reforms instituted under this program. A major concept underlying the design of this 
program is a capacity building effort with payoff beyond program the period of program 
implementation. As shown in Chapter 3, HPSGP schools performed, on average, no differently 
than non-HPSGP comparison schools, rendering the question of sustainability in the aggregate 
moot. However, individual schools may experience significant achievement gains under the 
program, and, the findings above show respondent perspectives of other improvements at 
participating schools that may not be fully reflected in student test scores. The final section of 
this chapter presents perspectives on sustainability of improvements made under HPSGP at the 
school level and from the district. 



Respondent Perspectives on Sustainability 

Our case study findings were fairly split on the issue of sustainability. Although individual staff 
at the same school did not necessarily share the same views, about half the sites generally 
expressed optimism in regard to their ability to carry reforms beyond the termination of the 
HPSGP and the other half were less, or not at all, optimistic. Across our broader survey sample, 
both school and districts generally held favorable views on the sustainability of school reforms 
and student achievement progress. 

Exhibit 6.5 presents a summary of the survey questions that assessed schools’ confidence levels 
about continuing improvement beyond the HPSGP grant. More than 8 in 10 school respondents 
believed that they would likely meet the API growth targets over the next three years. Nearly 60 
percent reported that the school would be able to continue improvement efforts beyond HPSGP, 
whereas a greater percentage of the district respondents (82 percent) shared this view. Contrary 
to these positive perspectives of sustainability, more than 60 percent of the school respondents 
noted that they were not able to replace HPSGP funds, or re-allocate current funds, to continue 
the reforms initiated under the program. 

On all three survey questions regarding sustainability, there appeared to be consistent and 
notable differences between the middle schools and other school types. In relation to their 
counterparts, middle school respondents were less likely to be able to replace HPSGP resources 
(70 percent in relation to an overall of 60 percent). These respondents also had a more doubtful 



This reflects schools that did not participate both in HPSGP and other state programs, such as II/USP and CSR. 
Including jointly funded schools, the total HPSGP funds amounted to $740 million between 2002-03 and 2005-06. 
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outlook, with only half believing that the sehools would be able to eontinue reforms beyond the 
program in eomparison to over two-thirds of high sehools. 

Exhibit 6.5. Survey Responses to Questions of Sustainability of Reform and 



Improvement 

Source: HPSGP School and District Surveys (AIR) 
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We were able to replace HPSGP funds or reallocate current funds to continue the reforms initiated under the 
HPSGP. 
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While a high degree of eonfidenee in sustainability eombined with the pereeived limited ability 
in seeuring replaeement funds may seem eounter-intuitive, spending patterns under the HPSGP 
may help explain why. For example non-personnel expenditures (e.g., teehnology and materials) 
are easier to maintain and will likely last beyond the life of the program. Longer term 
eommitments involving personnel, on the other hand, require eontinuing resourees. Indeed, a 
greater pereentage of sehools that were optimistie about their ability to fully eontinue reforms 
(40 pereent) deliberately spent HPSGP funds primarily on one-time or short-term items, in 
eomparison to less eonfident sehools (24 pereent). Nearly three-quarters of sehools that were 
eoneerned about sustainability spent HPSGP funds on release time for staff eollaboration or 

no 

professional development aetivities, eompared to less than 45 pereent of the eonfident sehools. 
Similarly, over a third of the less eonfident sehools spent funds on supporting teaehers’ time to 
provide additional instruetion, sueh as before sehool and Saturday programs, while less than 24 
pereent of the other sehools reported this as one of their top three expenditures. 

Strategies sueh as eollaboration, additional instruetional time, and more personnel are more 
ehallenging to eontinue in absenee of ongoing finaneial support, as evident from one sehool 
respondent, “We were foreed to displaee resouree personnel who were providing invaluable 
support to our students due to the diseontinuanee of the HPSGP, ineluding intervention 



While the survey question provided “materials” as an example of a short-term item, the meaning of one-time or 
short-term items was open to respondents’ interpretation. 

Based on survey question 16, which asked respondents to report the top three largest HPSGP expenditures. 
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programs, supervision staff during and after school, instructional assistants to help students 
during instructional time and for English Learners.” Another reported, “The extra personnel for 
class size reduction and remediation courses, along with money for staff development made the 
difference. However without the funding we’ve had to cut back in those areas.” Several schools 
provided similar examples of personnel cuts, and districts also expressed a reduced capacity to 
maintain personnel-related strategies beyond the grant’s tenure. For instance, despite the benefits 
of a longer school day for its English learners, one district returned to a regular school day due to 
insufficient fiscal resources. Retaining additional personnel secured through HPSGP funds has 
been made more difficult by districts’ declining enrollments, which has reduced their general 
funds. 

As HPSGP is a short-term funding program, the financial burden of maintaining reform efforts 
lies with the schools and districts after four years. Indeed, guidance in the application package 
for the first cohort explicitly stated that the CDE discourages using HPSGP implementation 
funds to hire teachers for class size reduction, presumably because it would be difficult to 
continue this effort beyond the duration of the program. 

Although hiring new teaching staff was discouraged, spending on supplemental personnel was 
reported as the most effective expenditure by school respondents, followed by staff release time 
for collaboration and supporting staff time for additional instruction, as shown earlier in Exhibit 
5.12 in Chapter 5.^°^ This is troubling, as it suggests the strategies perceived as most effective for 
these types of schools are also the hardest to sustain without ongoing financial support. The 
dilemma of investing in effective strategies only to have them cease with dire effects was 
described by a school respondent: “Effective positions, such as our very successful parent 
liaisons, were summarily cut as soon as the major portion of our funding ended, and our parent 
interest plummeted accordingly.” Another district respondent also observed a decline in student 
achievement after school personnel were let go after the grant ended. 

However, school respondents’ positive views on sustainability do not appear to be strongly 
related to their prior progress. Schools that did not make their schoolwide and comparable targets 
in 2005 were only slightly less likely as those that did to believe that they would meet their 
targets in the next three years (86 percent versus 93 percent). Moreover, these results were 
similar when examining schools by their annual growth rates in CST math and ELA, averaged 
across 2003 to 2005, with the higher performers only slightly more likely to agree that they 
would meet the API targets. This is in line with our case study findings, in which confidence in 
sustainability did not differ appreciably between schools shown to be experiencing success under 
this program as opposed those not showing success. This suggests that some schools may have 
an unrealistic view of continued progress, or more specifically, schools that showed little 
academic progress previously may believe it is easy to continue to make minimal improvement. 



Among districts with HP Only schools (n = 95), 48 percent show declining enrollment from 2001 to 2005; the five 
districts with the largest number of HP Only schools observed an average 4 percent decline in enrollment during that 
time period. 

The cohort 1 HPSGP guidelines read: “The district must bear in mind that they will be accountable for the 
achievement benchmarks delineated in the approved Action Plan. The CDE discourages using HPSGP 
implementation funds for [class size reduction] purpose.” 

**** Based on responses to a survey question asking for the top three most effective expenditures. 
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District Roie in Sustaining Reform 

Nearly 84 percent of school respondents expressing confidence about continuing 
reform characterized their districts as being highly or reasonably supportive, 
whereas only 63 percent of respondents from the less confident schools did so. 

In addition to investing in shorter-term items, sehools more eonfident about sustaining reform 
may be reeeiving additional support from the distriet. As one sehool respondent noted, “The 
impact will last because we have invested in both materials and professional development and 
support, which will continue with the support of leadership at the school site and at the district 
level.'" The respondent subsequently wrote that the district office was “extremely supportive and 
provided clear guidance on HPSGP.” Nearly 84 percent of respondents confident about 
continuing reform characterized their districts as being highly or reasonably supportive, whereas 
only 63 percent of the less confident schools did so. Conversely, 17 percent of the less 
confident schools described their district as failing to provide support or seriously hurting 
improvement efforts, in relation to 5 percent of the confident schools. This finding is further 
supported by the district responses. More than 40 percent of districts that described themselves 
as focused on school improvement with coherent goals strongly agreed that the schools would 
continue reforms, whereas only 4 percent of the other districts did. 

As mentioned previously, more than 80 percent of all surveyed districts believed that their 
schools would fully continue the reforms. While three districts appeared to place the onus on the 
schools (e.g., “the schools will need to be creative in terms of finding resources to continue 
them”), about half of these districts pointed to evidence that improvement efforts were ongoing 
with support from the district. Even though they agreed that schools would sustain reforms, some 
districts acknowledged that fiscal constraints prevented them from fully implementing strategies. 

Irrespective of their confidence in the continuation of reforms, nearly all district respondents 
identified some key district supports in an attempt to provide some degree of continuity, 
including professional development (39 percent of the respondents), coaches and other resource 
staff (37 percent), and assistance with data analysis/monitoring (29 percent). Other less 
frequently identified district supports included supporting principals (through support teams and 
monitoring); budgetary and planning guidance; and prioritizing resources to lower performing 
schools (in terms of data and staffing). 

While not consistently specified across all respondents, these district supports were available for 
all schools in 13 districts, while 9 targeted HPSGP or lower performing schools. Nearly a quarter 
of the districts planned to supplement the schools’ loss of HPSGP resources by seeking 
additional funding (e.g., grants, particularly the Quality Education Investment Act) or re- 
allocating existing categorical/general funds. More than 20 percent of the districts attributed 
larger reform efforts and other programs, such as SAIT, Reading Eirst, and most commonly 
Program Improvement, for helping maintain the momentum. 



These figures exclude charter school responses. Significant at the 5 percent confidence level. 
These figures exclude charter school responses. Significant at the 10 percent confidence level. 
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When asked about the challenges of sustaining improvement, respondents most often reported 
limited funding, with some districts reporting a further financial hit from declining enrollments. 
One district lamented, “You can’t infuse something with money, take it away, and expect things 
to continue. You need constant infusion of cash in a low wealth area. We are limping along 
trying to do the same things. We have tried to get funds from other areas, but will our 
improvement efforts be as robust? No. We have the goals, but we don’t have the money.” Added 
to this was the difficult task of maintaining focus and positive attitude for overworked, exhausted 
teachers and administrators. Recruitment and retention of high quality staff, particularly in light 
of staff attrition, further taxed districts’ capacity to sustain improvements. 



Summary 

This chapter summarized school and district perspectives on the importance, impact, and 
sustainability of the HPSGP. Overall perspectives on all three of these measures were positive on 
average. Respondents believed the program had been important to their school’s improvement 
efforts; that it had an impact in such important domains as teacher knowledge, resource levels, 
and academic performance; and that they would be able to sustain achievement gains over time. 
In response to the survey’s last open-ended question which asked respondents if there was 
anything else they wanted to share about the HPSGP, 30 percent expressed their unsolicited 
appreciation for the program. Clearly, respondents believed the HPSGP had made a difference 
for their schools. 

This overall sense of optimism, however, must be considered within the context of our other 
findings. These include the negligible (and essentially non-educationally significant) program 
effect observed in Chapter 3; the high percentage of schools that reported they were unable to 
replace HPSGP funds; and some notable concerns regarding breakdowns in program 
implementation reviewed in Chapter 5. 

As an example of this possible contradiction, after indicating agreement that their school would 
be able to meet their API growth targets in the next three years, in their follow-up narrative 
response several school respondents clearly indicated concerns about loss of funding and 
sustainability. As one explained, “Once the funding is discontinued, so are the programs and 
services that were funded by High Priority. The challenges of working with at-risk students 
continue even though the funding does not.” 
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Chapter 7: Summary of Findings and 
Recommendations 



This concluding chapter begins with a general discussion of observations and findings, 
followed by recommendations. This latter section starts with HPSGP-specific 
recommendations provided by district and school respondents to our Year 2 survey 
efforts. It concludes with overall and program- specific recommendations from the study 
team based on information derived from the multiple data collection and analysis efforts 
of this study as described in Chapter 2. 



Observations and Key Findings 

The following general discussion starts with our general observations and selected 
program-specific findings resulting from this two year evaluative effort. 

General Observations 

Given the primary purpose of the HPSGP, some may consider the only relevant finding 
to be the one that showed no substantial difference in student performance between 
HPSGP and comparison schools. Overall performance of low-performing schools (both 
those participating and not participating in the HPSGP) is improving in an era in which 
state and federal accountability systems have been introduced. The accountability 
movement, including interventions like the HPSGP, has cast an important spotlight on 
chronically underperforming schools. An expectation is being conveyed to state, district, 
and school administrators that the status quo for these schools is no longer acceptable. 

This increased attention paid to the state’s lowest-performing schools is laudable, and has 
yielded some positive results for these schools on average as well as for all schools 
statewide. State and federal accountability efforts have likely made a substantial 
contribution to this improved performance, and it seems likely that in a generic sense, the 
HPSGP has contributed to these overall gains as well. 

At the same time, analyses of school- and student-level achievement for this evaluation 
show no meaningful difference between schools participating in the HPSGP and 
comparison schools. Likewise, two prior evaluations of the II/USP (which was similar in 
many ways to the HPSGP) found that while the program focused attention on student 
achievement and low-performing schools, there appeared to be negligible overall impact 
on student achievement in participating schools (O’ Day & Bitter, 2003; Bitter et ah, 
2005). 



American Institutes fer Research 



Page 1 45 





Evaluation of the High Priority Schools Grant Program 



The II/USP and HPSGP efforts to shore up performance in the state’s lowest performing 
schools are well-intentioned. At the same time, it also must be acknowledged that these 
programs (which in many ways are quite similar) constitute a substantial state investment 
over a considerable period of time with no discernable differential gains in student 
performance at target schools on average. Given these consistent findings, the state may 
wish to consider alternative approaches to improving student outcomes in low performing 
schools. 

The vast majority of our district and school survey respondents and nearly all of the sites 
we visited clearly appreciated this program. At some of these locations, it was cited as a 
primary basis for improving student performance. On the other hand, the best empirical 
evidence we have does not show participating schools in the aggregate as making 
substantial gains. While we observe growth in student performance in HPSGP schools, it 
does not differ in a meaningful way from the performance of schools not in the program 
(i.e., not receiving a supplemental $400 per year per student over three to four years). 

Perhaps the major question arising from these findings is why such a large investment 
shows no appreciable results. In addition, it seems important to ask whether the program 
should be more rigorously implemented as is, should be changed in modest ways, 
radically altered, or dropped altogether. 

One argument is that interventions like the II/USP and HPSGP are simply supplements to 
an overall system that is fundamentally flawed. Even though the fiscal investments 
associated with these interventions are fairly substantial, because they are partial, short- 
term solutions attempting to overcome much more substantial system flaws, they have 
relatively little chance of effecting measurable progress in low performing schools. 

The contention that categorical programs like the HPSGP have little chance of making 
substantial impact given the fundamental areas of weakness of the overall state system 
was a major conclusion of a recently completed evaluation effort of the state’s K-12 
public education system. Funded by a consortium of foundations, and led by Professor 
Susanna Loeb of Stanford University, this collection of studies - known as Getting Down 
to Facts - was conducted by request of state leaders at the highest levels, including the 
Governor, the leaders of the Senate and Assembly, and the State Superintendent of Public 
Instruction. 

A summary paper resulting from these studies (Loeb et ah, 2007:1) starts with an 
overview of the problem: “On many different measures of achievement, California’s 
students fall far behind those in other states.” After reviewing the considerable body of 
evidence amassed through this comprehensive collection of studies, the authors further 
conclude (2007:5): 

There is no evidence to support the idea that simply introducing yet more new 
programs will produce the desired achievement gains. California already has far 
over 100 well-intentioned categorical programs, and there is no reason to think 

*®"^These included the The Bill and Melinda Gates Foundation, The William and Flora Hewlett Foundation, 
The James Irvine Foundation, and The Stuart Foundation. 
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that adding one or two more will make much difference, no matter how carefully 
targeted or lavishly funded. The marginal impact of any new program will be 
small. Quite simply, the finance and governance system is broken and requires 
fundamental reform not tinkering around the edges. 

If this assessment is correct, it would explain the lack of observable HPSGP impact and 
would further suggest that additional iterations of limited program interventions like the 
HPSGP, regardless of the degree to which implementation is improved, will not likely 
substantially impact the state’s low performing schools. 

An alternative perspective is that substantial gains can result from programs like the 
HPSGP with some redesign and shored up implementation. For example, the relatively 
short-term design underlying the HPSGP may be one reason for its lack of impact. That 
is, a three- to four-year funding cycle may be insufficient to substantially affect school 
performance. Other possible reasons include implementation breakdowns, such as weak 
external providers, short-lived action plans, insufficient district involvement and support, 
excessive principal turnover, and constraints related to the sometimes untimely arrival 
and use of HPSGP funds. 

The case studies left the site visit teams with the impression that the HPSGP resources 
were making a difference in schools where the types of implementation issues discussed 
throughout this report did not interfere. Through the case study site visits and surveys, 
there was clear evidence that the program worked well at individual school sites. 
Reasonable continuity of capable leadership and staff in the school appeared to encourage 
the long-term planning and implementation that is needed for supplemental funds of this 
type to be strategically spent. Under these conditions, it appeared that schools were able 
to use HPSGP resources to purchase combinations of personnel, non-personnel, and 
contracted services (e.g., external training or conference participation) to make a 
substantial difference in the academic experience and outcomes of students. Given the 
temporal nature of these resources, however, their ability to sustain these results is open 
to question. 

A clear majority of the survey respondents perceived that the program had made an 
impact. However, there was also a fairly sizeable percentage of respondents indicating 
implementation concerns that could substantially impact the observed lack of a program 
effect in the aggregate. For example, over 40 percent of school respondents indicated that 
the district had not actively assisted them in regard to program implementation. Half of 
the school respondents expressed concern about the length of the program and a third said 
that funding had not arrived in a timely fashion and therefore did not fully permit 
appropriate planning and spending. Considerable HPSGP funds were carried over each 
year, which was often described as reflective of other systemic implementation break 
downs. 

Using survey data, we attempted to test the general contention that more rigorous 
implementation would result in improved outcomes. However, examining the 
relationship between the perceptions of survey respondents and student outcomes did not 
yield clear results. While we believe that better implementation would increase the 
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potential of the program making a difference - and a substantial number of survey 
respondents (although a minority) also indicated implementation concerns - we simply 
could not document this based on available data. 



Listing of Seiected Program Findings 

• On average, HPSGP schools showed gains in student performance during the 
period of program implementation. However, the differences in student 
performance between the schools participating in this program and comparison 
schools across a variety of comparative analyses were negligibleAlthough more 
than 60 percent of school respondents reported that a plan for school improvement 
prominently guided their reform efforts, this reported impression was not 
reflected in measured academic gains. 

• While nearly half of the school respondents described their external provider 
support as appropriate and effective, nearly 45 percent of the surveyed 
respondents reported that the school did not use, or reported that they did not 
know if the school used, an external provider in the development of the Action 
Plan, even though it was a program requirement. 

• Although the HPSGP attempted to more clearly define the district role, over 40 
percent of school respondents indicated that the district had not actively assisted 
them in the implementation of the program. 

• Although the vast majority of HPSGP school respondents indicated an effective 
use of funds, half expressed concern about the short length of the program and 
nearly a third reported that the untimely arrival of funds did not permit 
appropriate planning and spending. 

• Approximately one-quarter of HPSGP funds were carried over each of the four 
implementation years. 

• Spending on personnel was reported as the most common as well as the most 
effective use of HPSGP funds. 

• HPSGP was perceived as having a major role in student achievement gains, 
despite nearly identical academic performance during this period between HPSGP 
and non-HPSGP comparison schools. 

• A slight majority of respondents reported a lasting HPSGP impact in areas of 
school capacity. 

• While 60 percent of school respondents indicated confidence in sustaining the 
impact of HPSGP, only 40 percent said they had been able to find funding to 
continue these reforms. 

• Nearly 60 percent of surveyed school respondents cited major factors limiting 
their ability to use HPSGP funds to improve student achievement at their school. 
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Recommendations 

The following section reviews key program recommendations made by the school and 
district respondents surveyed for this evaluation. These are followed by study team 
recommendations, which are categorized into general state-level recommendations, 
irrespective of the future of the HPSGP, and specific improvements to the program that 
may foster a greater impact. 



Respondent Recommendations 

As part of the narrative section of the school survey, school respondents were given an 
opportunity to recommend a single most important improvement to HPSGP. These open- 
ended responses fell into six broad categories, as shown in Exhibit 7.1. Among the 80 
school respondents who answered, most made recommendations relating to the design or 
structure of funding under the program. The reader should note that while the question 
specified a single improvement, most respondents provided more than one response. 



Exhibit 7.1. School Respondents’ Recommendations to Improve the HPSGP 

Source: HPSGP School Surveys (AIR) 



What is the single most important improvement to the HPSGP that you would recommend to state 
policymakers? 


Theme 


N 


% 


Longer Term Funding 

Examples: Provide lower level of annual funding over longer time period 
Gradually reduce funding levels to facilitate budget/ strategic adjustments 
Provide continued funds to successful schools 


36 


34% 


Timely Funding and Consistent Carry-Over Policies 


15 


14% 


Increase Flexibility in Spending 


15 


14% 


More Explicit Funding Guidance or Required Uses 


12 


11% 


External Provider-Related 

Example: Allow schools to opt out 


6 


6% 


Dissemination of Knowledge / “Best Practices” 


4 


4% 


Other 


18 


17% 


Total Responses 


80 


- 



Note: Percentages are based on 106 respondents, and add up to more than 100% as respondents provided more than 
one answer. 



Nearly 35 percent of the school respondents recommended that funding under the HPSGP 
be provided for a longer period of time, reflecting the idea that reform takes, in addition 
to money, time: “Three or four years is too short a time period to make permanent 
changes. The funding was a short-term solution to a long-term problem.” How and under 
what conditions respondents proposed continuing funding varied. Six school respondents 
favored the idea of diluting the yearly funding amount in order to stretch the money over 
a longer period of time. One respondent underscored the importance of self- 
sustainability, saying that a gradual reduction of yearly grant awards could help schools 
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phase in alternative funding. Others recommended that continued funding should be 
coupled with specific performance benchmarks in order to both reward “successful” 
schools and ensure the viability of their programs. In addition to these 36 school 
respondents, 7 districts also noted concerns with the short length of the grant period. 

Fourteen school respondents as well as 7 district respondents expressed a need for more 
clarity around funding, primarily consistent and predictable funding allocation timeline 
and carry-over requirements. One school response deserves special note, as it effectively 
captures the concerns expressed by others: 

The one recommendation I would make is for the state to be very clear 
about its funding and to commit to providing the funding in a timely way. 

Money given after the fact does not facilitate thoughtful planning on the 
district or the school levels. Resources, supports, and appropriate 
trainings cannot be planned for and implemented if funding is not given 
until the last minute. 

Six respondents from year-round schools stressed that the delivery of funds 
needed to better align with the school calendar. 

Another recommendation pertained to governance over the use of funds. Fourteen school 
respondents cited that increased flexibility and decision making authority over how funds 
are spent would facilitate reform efforts. In most cases, these recommendations were 
based on the notion that school sites are most attuned with student needs and therefore in 
the best position to determine a spending plan customized to those needs. Three of these 
responses suggested that the allowed uses of the funds were too narrow, with respondents 
often citing specific expenditures that would have been beneficial if they had been 
allowed (such as facilities, school safety and bilingual materials). Lastly, three of these 
respondents mentioned excessive district control and bureaucracy as a reason for 
enhancing site-based authority over spending decisions. 

On the other hand, a different 14 school respondents recommended that clearer, more 
targeted goals and specific requirements be associated with the grant money. Responses 
under this category often recommended setting specific spending requirements or 
strategies such as class size reduction, technology, opportunities for extended learning 
and collaboration, among others. 

On a related topic, four school respondents recommended broader dissemination of best 
practices and successful strategies in HPSGP schools, including research-based 
interventions. One in particular suggested “a conference or gathering for HPSGP schools 
to share successes so we can learn about different ways to improve student achievement.” 

Eight district respondents pointed to a lack of state guidance on best practices and 
technical assistance as a weakness of the program. Six of these districts recommended 
that the state build knowledge of effective school reform models, with particular 
emphasis on what works for districts. Two of these districts described examples of 
research-based strategies they had learned about at conferences - information that they 
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thought would be helpful for the state to disseminate on a statewide basis, as opposed to 
having to seek out the knowledge through other means. Another district respondent 
remarked, “We spend too much time roaming in the unknown.” 

In making recommendations to improve the program, five school respondents and nine 
district respondents provided feedback on the external provider component. One of the 
school respondents noted a need for more clarification regarding the responsibilities of 
the provider, and three others called for a reduced role by allowing schools to opt out of 
the requirement or focusing on building capacity within the school instead. 

Of the nine district respondents mentioning this as an area for improvement, six proposed 
more guidelines in the selection of eligible providers, including greater quality control 
and screening. One in particular recommended an evaluation of provider quality. Based 
on its mixed experience, one district learned the importance of carefully selecting the 
providers, “There is some un-evenness in how prepared these external providers are to 
really provide the level of support the schools need.” Four districts sought more 
prescriptive roles for the external provider, such as working regularly with the schools; a 
long-term involvement for the duration of the grant, and district-level involvement. 
Describing a school whose key program efforts ceased at the end of the grant, one 
respondent stressed the importance of including district in the external provider work to 
help sustain reform. 

Several of these provider-based recommendations are reflected in the new cohort 2 
expectations, which require regular communication between schools, districts, and 
external providers, as well as ongoing external support to both the school and the district 
in the implementation of the plan. Although it was not identified by school respondents 
as an area for improvement, seven districts recommended bolstered state oversight and 
monitoring, including interim progress reports (with three districts citing the SAIT 
structure as a framework for this). 

Recommendations from the Study Team 

These recommendations are divided into two major categories: I) overall state 
recommendations and alternative investments in lieu of programs like HPSGP, and 2) 
HPSGP-specific recommendations, some of which reflect the recommendations made by 
the survey respondents. In keeping with the discussion at the onset of this chapter as to 
whether the failure of the HPSGP appears more due to global limitations of K-12 
education in the state overall as opposed to programmatic issues, the first set of 
recommendations takes more of a global perspective. What changes might be made to the 
overall state system to enhance student performance in low performing schools? These 
recommendations are more general and are not specifically limited to the HPSGP or 
similar programs. Nevertheless, they are drawn largely from what has been observed and 
heard from site respondents throughout this study, as well as lessons of successful school 
practices from the IFUSP evaluations, e.g. the value of data driven decision making. In 
addition to these suggestions which were specifically generated as a result of our 
evaluation of the HPSGP, we again refer the reader to the Getting Down to Facts findings 
and their implications in regard to overall state K-12 education reform. 
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Second, we make recommendations regarding future implementation of programs like the 
HPSGP. Should such programs continue, these recommendations are intended (but 
certainly not guaranteed) to increase the chances of future program success. They include 
increased assurances of district involvement and accountability in the form of at least 
equal resources at targeted schools before adding HPGSP funds and the provision of 
stable site leadership as preconditions to participation. They also include provisions 
designed to make the Action Plan and the external provider components of the HPSGP 
more effective. 

General State-Level Recommendations 

Based on our findings, it appears that a short-term categorical approach to school reform 
is insufficient to overcome much larger system inadequacies that fail to provide the kinds 
of long-term support and assistance needed to substantially and consistently improve 
student performance in the state’s most challenged schools. We suggest terminating 
categorical interventions like the HPSGP in favor of more comprehensive statewide 
school reform that provides long-term administrative and resource support to the state’s 
lowest-performing schools enrolling our most academically challenging students. 

However, we also understand broad-reaching state reform to be an unlikely immediate 
alternative. In the interim, we recommend that the state consider alternative investments 
to bolster the performance of the state’s lowest-performing schools, as opposed to relying 
on II/USP- and HPSGP-type interventions. 

For example, the most fundamental element supporting effective reform efforts at the 
schools we visited was constancy and strength of school leadership. To provide the 
state’s most challenged schools a reasonable chance to successfully compete with other 
schools in the state (and across the nation), it seems imperative as a first step that we test 
alternative methods for attracting and retaining some of the best instructional staff at 
these sites. Because it will be difficult to know with any confidence at the onset that any 
of these strategies will show a measurable impact on student performance, we 
recommend trying several of these alternatives in different parts of the state with a 
corresponding evaluation design to see which ones seem most productive and worthy for 
consideration of replication on a larger scale. 



1. Keep the attention on student learning and low-performing schools. 

State and federal standards-based policies have been very successful in capturing the 
attention of the education community and the general public and focusing that attention 
on student outcomes system-wide and on low-performing schools in particular. We urge 
that this attention continue. 

2. Consider the resources needed for sustained academic success in low- 
performing schools, and ensure that they are present and maintained in 
these schools and their districts. 

The state should identify the resources needed in the state’s most challenging, highest- 
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poverty schools, fund them accordingly, and ensure that these resources are allocated 
effectively by districts to schools. 

Since the district was found to be a key intermediary between state-level policy and 
school-level implementation, the state should ensure that districts have the resources to 
provide the necessary assistance and support to their schools, and that they allocate them 
to low-performing schools as needed. For example, they might be required to ensure a 
teaching force in these schools that at least equals, or exceeds, the district average on 
such characteristics as experience and degree level. 

Three of the studies included in the Getting Down to Facts project described above 
systematically considered the funding needed for public education given the state’s 
specified academic goals and the composition of student needs in districts across the state 
including students living in poverty, English learners, and students with disabilities. 

These studies provided alternative estimates of the resources needed on an ongoing basis 
to meet the state’s educational goal, and concluded that money alone would be 
insufficient to enhance educational outcomes in the state. However, all studies concurred 
on the need to target enhanced resources in schools with the most challenging 
populations on an ongoing basis. These clearly include schools currently targeted for 
such temporary interventions as HPSGP. 

3. Use data on an ongoing basis to identify the extent to which state-ievei 
programs make an impact, and use these data to inform and aiter state- 
ievei poiicy and programs in support of tow-performing schoois as 
needed. 

As the state sets expectations for schools and districts and encourages them to regularly 
use data as a basis for shaping policy and practice, we suggest the same process for the 
state. The state should attempt to actively determine fairly early on how well state- 
supported interventions are working. 

We recommend early and rigorous assessment of a formative nature that can serve to 
guide and adjust implementation, and that is designed to compile evidence as early as 
possible about the extent to which anticipated outcomes are likely to be forthcoming. 
External evaluations provide a means to gain formative and summative information on 
programs. However, given the high-stakes environment and urgency to improve student 
outcomes, the state itself should establish more mechanisms to review policies regularly, 
assess what components of its policies are on the right track, and adjust policies on an 
ongoing basis as needed. 

4 . Enhance the power of CBEDS 

Several of our case study sites and survey schools exhibited alarming principal turnover, 
and teacher turnover was also noted as a particular challenge to reform efforts. However, 
we were not able to compare this reported turnover to our designated comparison schools 
or other groups of interest, as this critical information is lacking in CBEDS. We 
recommend enhancing CBEDS to include questions on the number of years that 
principals and teachers have been at their current school, and the number of years in that 
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same position in other schools. This enhancement to the database would serve as a 
powerful tool to understand staff turnover and its implications for student achievement. 

5 . Foster data-driven decision making. 

Many of the successful schools we have examined (through evaluations of the HPSGP, 
Proposition 227 (Parrish et ah, 2006), IPUSP (O’Day and Bitter, 2003; Bitter et ah, 

2005), and high poverty schools (EdSource, 2006)) at least partially attribute this success 
to the regular assessments and review of data to drive instruction. Many of these systems 
were said to be locally developed. The state may want to encourage broader development 
and dissemination of such systems in districts and local schools. Unlike a number of 
other factors that have been repeatedly cited as making a difference in regard to school 
reform (e.g., strong leadership), data-driven decision making may be much easier to 
replicate. 

6. Recognize the infiuentiai rote districts ptay in facititating or 
constraining schooi improvement, and incorporate mechanisms into 
accountabiiity poiicies to encourage positive and productive actions at 
the district ievei. 

One of the key findings of this study was the potential influence of district context on 
schools’ achievement growth. Although the HPSGP attempted to increase the 
involvement of the district in these reform efforts in relation to the lEUSP, the findings 
from this study show that there is considerable room for improvement. The state’s 
District Assistance Intervention Team (DAIT) process should further clarify the role of 
districts and counties in regard to assisting the state’s lowest-performing schools. 

7. Consider methods to better aiign the state and federai accountabiiity 
systems. 

Site-level respondents in this study largely reported that while they consider the API to be 
a better outcome measure, they feel pressure to address AYP targets. This is not 
surprising given that 80 percent of the HP Only schools are in Program Improvement 
(PI), with nearly half of the PI schools in Year 4 or 5 of the sanctions. Given the conflict 
and confusion associated with two overlapping accountability systems, we recommend 
that the state focus further on their alignment. However, we recognize that this is not an 
easy task and should be done carefully to preserve the most effective aspects of the state 
system. 

For low-performing schools that are far from reaching the AYP targets set through 
NCLB, API growth targets provide incentives for schools to make continuous 
improvement. These targets account for the fact that these schools are starting from a 
very low base of performance. 

Potential actions the state can take include examining better ways to align the 
expectations and associated sanctions of the state model with the federal model. For 
example, the state could focus on having the same schools identified under both programs 
for similar sanctions. Having some schools recognized for success within one model, but 
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designated as failing in another, can result in confusion and unclear expectations. 
Additionally, the state can continue to press the federal government for greater 
incorporation of the state API into the AYP measure. Since there appears to be greater 
buy-in to a growth model and to the API measure in general, this may be an important 
aspect of the state model to attempt to preserve. 

8. Develop and foster policies that will strive for strength and continuity of 
school leadership, especially at low-performing schools. 

Through the site visits and survey data, we have documented the common problem of 
excessive turnover in the leadership at low-performing schools. Conversely, where 
schools have appeared to thrive under these types of interventions, strong and ongoing 
school leadership was commonly found to be an integral part. While change in leadership 
may be the catalyst necessary to meaningful change, it appears very difficult for 
meaningful long-term planning and change to take hold without subsequent stability of 
leadership. We believe that a valuable role for the state, and a possible alternative 
investment to the HPSGP, would be to allocate funds for recruiting, training, and 
retaining strong principals in our state’s most challenging schools. 

9. Work with districts to develop learning networks where districts and 
schools in need of improvement can be linked with, and can learn from, 
districts and schools that have been successful in improving outcomes 
with comparable populations of students. 

In light of the limited communication reported and evident among schools participating 
in our evaluation, we recommend that the state and districts consider working in tandem 
to create opportunities for districts and schools to learn from one other. This could 
enhance knowledge transfer from schools showing substantial progress under reform 
efforts over time to schools new to and struggling with reform. Such learning networks 
might feature pairing of schools (“sister” schools) or clusters of schools that would 
collaborate and work together toward the common goal of enhancing student 
achievement. 

Although it is likely that costs will be associated with this kind of system (e.g., stipends 
to staff from successful schools offering assistance to similar, struggling schools), this 
may prove a more effective use of state funds than such mechanisms as external 
providers. It seems reasonable to expect that the state’s greatest talent pool in regard to 
turning around low-performing schools resides in the administrators and staff who have 
demonstrated that this can be done. 

Similarly, the state should identify districts that have successfully improved student 
outcomes in their low-performing schools, and facilitate opportunities for other districts 
to learn from them. Given that districts can play a key role in improvement efforts at their 
schools, and influence the implementation of state-level programs like HPSGP, we 
recommend a similar effort as above, but targeted at districts. In this case, the state would 
again set simple and straightforward criteria for high-growth districts that can be 
reviewed on a yearly basis. Using these data, we recommend that the state provide 
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opportunities, either through regular conferences and meetings, or through a more 
intensive partnering program, to facilitate learning across districts. 

It has been reported that the state Title I office previously proposed matching their 
“Achieving Schools” with struggling schools, but the successful schools did not want to 
distract from their schools reform efforts by having key personnel districted and 
frequently off site. In research we have conducted in the past, it also has been our 
experience that some of the most successful districts we have identified in regard to 
demonstrating that they can shore up schools with challenging student populations 
sometimes discourage site visits and other opportunities to share what they are doing. 
Much of their success is likely associated with deterring these kinds of distractions while 
maintaining a strict focus on their primary goals. 

With fiscal incentives, however, successful schools and districts may very likely be 
willing to reconsider and see this as an opportunity to retain the resources they need to 
maintain their reform efforts through sharing what they have learned with others. We 
believe the costs associated with this kind of system (e.g., stipends to staff from 
successful schools offering assistance to similar, struggling schools) is a possible 
alternative use of state funds the state may wish to consider in shoring up struggling 
schools and districts. It seems reasonable to expect that the state’s greatest talent pool in 
regard to turning around low-performing schools resides in the administrators and staff 
who have demonstrated that this can be done. 

10. Look at other states’ efforts to support their lowest-performing schools. 
Assess what investments they are making toward these ends and the 
degree to which they are experiencing results from these efforts. 

As a result of national and state accountability systems across the country, many states 
are experimenting with interventions with the same basic intentions as the HPSGP, i.e., to 
improve performance in their most challenged schools and districts. We suggest an 
investigation into what other states are doing and what evidence they have found in 
regard to a return on these investments. 

1 1. Require participation in future evaluations. 

As a grant precondition for any state program, districts and schools should agree to 
participate in state-approved evaluations of the programs. Soliciting the participation of 
districts and schools for this study took considerable persistence. As the state makes 
substantial investments in programs of this type, a reasonable pre-condition for 
participation is the state’s right to collect data regarding whether this investment is cost 
effective. 

Specific HPSGP Recommendations 

If the decision is made to continue with interventions like HPSGP, we suggest continued 
efforts to address some of the implementation challenges outlined in this report. 
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1. Target “failure” early: The CDE should monitor the performance of HPSGP 
schools annually and identify actions for schools that do not meet their 
API growth target in a given year. 

When schools are not showing progress annually (e.g., they do not meet their API growth 
target in a given year), there should be an increase in oversight, such as requiring 
ramped-up support from the district and possibly a required continuing role for the 
external provider. Conversely, when schools are showing progress, it may be advisable to 
add additional rewards, such as relaxed requirements (e.g., increased independence or 
flexibility to carry-over funds beyond the final year of the grant). 

Another possible component of the accountability provision listed above is that external 
providers might be required with input from the District/School Liaison Team (DSLT) 
and school site council members to issue a status report for their schools that did not meet 
their growth targets within four months of the release of the API scores. The report would 
detail factors that are preventing the school from progressing and list specific steps that 
need to be addressed by the school and/or district to overcome observed impediments to 
success at the site. In the interests of efficiency, the CDE may want to target HPSGP 
schools whose growth is “red” (e.g., zero or negative growth) in a given year. In the first 
year of implementation of the HPSGP (2002-03), only six schools made no or negative 
growth on their API; in 2003-04, 137 schools were “red.” While this rubric is not the only 
way to identify schools not making progress, and we encourage the examination of 
additional measures, we believe that it provides a reasonable starting point for 
considering which schools to target early in the process. 

After another year of not meeting the API growth targets, schools might be required to 
ramp up external support even more, possibly with a different external provider. Or, 
perhaps in these cases it would simply be more expedient to accelerate the SAIT process. 
Overall, however, it seems important to increase intervention, guidance, and support as 
early as possible for schools that are clearly not making expected progress through the 
HPSGP. It also seems important to convey a sense of accountability for the external 
provider, as well as the district, in regard to the school’s performance. They need to be 
seen as a team, jointly responsible and jointly accountable for school improvement. 

2. Enhance the district role: The role of the district should be explicitly 
enhanced and the district should be held accountable for school progress 
and for establishing and maintaining “conditions” for success. 

We recommend that bolstered assurances for which districts will be held accountable be a 
prerequisite for school participation in the HPSGP. The analyses from this study suggest 
that active engagement of districts is an important pre-condition for program success. 

This recommendation mirrors the guidelines developed by the CDE for the second cohort 
of HPSGP schools, which institutes a continuous improvement process facilitated by a 
District/School Eiaison Team. The guidance also calls for the Action Plan to demonstrate 
a clear support role for the district in the development and implementation of the plan and 
shared responsibility for school progress. 

In fostering district accountability, we recommend that the CDE develop a system of 
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rewards and sanctions at the district level that are associated with the success or lack 
thereof of participating schools. For example, in regard to the assurances above, district 
compliance should be especially closely monitored in cases where participating schools 
are not showing success. Initially, districts should be reminded of their responsibilities in 
regard to program implementation and that these assurances must be fulfilled to allow 
continued program participation. Ultimately, if districts do not comply and schools are 
continuing to fail, ongoing program funding should be withheld. Rewards for gains in 
student performance might come in the form of increased local discretion. 

As mentioned in this report, in some instances certain pre-conditions for successful 
program implementation appeared lacking. Capacity building at these schools must be 
considered a district priority. The types of pre-conditions for application could include 
additional assurances that applying schools will receive assistance from the district in the 
following areas: 

Assurance 1: Applying schools are already at, or preferably above, the district average in 
regard to levels of personnel and non-personnel resources, or will be before the end of the 
first year of implementation. Extant state data could be used to develop indices measuring 
this. 

There is already a district assurance in the original HPSGP application requiring that the 
percentage of fully credentialed and experienced teachers at the school increase at least to 
the district average by the end of the second year of implementation. According to this 
assurance, the increase after the first year of implementation will be at least one half of 
the total increase needed. It is important that personnel beyond teachers be included (e.g., 
administrative and support staff), and that a non-personnel resource equity measure be 
added. The resource allocation analysis completed for this report might provide some 
basis for these personnel measures. 

Assurance 2: Districts should also assure that schools have, or will be assigned, a 
principal with some evidence of prior school success. It would be incumbent on the 
district to provide such evidence; some waivers may be available for rural and/or small 
districts. 

Assurance 3: The district should also take steps to ensure reasonable continuity of staff 
during the grant period. Principal and teacher turnover was inordinately high at many of 
the case study schools, and an analysis of principal experience suggests that about 30 
percent of HPSGP principals have been at the school site for less than one year. This 
continual disruption in staff was often cited as a major challenge for establishing and 
progressing towards a clear vision. Districts might develop and institute policies and 
programs to encourage stability, such as financial or professional development incentives. 

Assurance 4: HPSGP Schools will be favored over other schools with regard to selecting 
replacement staff in the case of personnel openings (e.g., an HPSGP school would 
receive first choice for a literacy coach opening). 
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3. Improve monitoring: The CDE (perhaps with the assistance of the County 
Offices of Education) should enhance its monitoring of non-achievement- 
related measurements, such as compliance with the district assurances 
and expenditures. 

Along with these district assurances, we recommend regular reporting and monitoring. 

As the CDE is charged with allocating HPSGP funds, they should also be given the 
responsibility and authority to ensure that the program is implemented as designed and to 
terminate the program in a given school or district- wide when this is clearly not the case. 
In order to ensure that districts do not lose sight of these obligations, the Annual Reports 
should include data that will enable districts and the CDE to assess progress towards such 
assurances (e.g., districts should report the percentage of fully credentialed teachers at the 
district and for each of its HPSGP schools). Districts that are not showing progress within 
the expected timeframe should be required to provide a brief report to the CDE on what 
steps the district will take to address these discrepancies. In short, there should be some 
degree of state monitoring of compliance with measurable agreed- to assurances. 

This monitoring process could also include flagging schools for review if they under 
spent the annual grant by more than 50 percent. In our case studies, under spending was 
usually an indication of other systemic problems, such as a high degree of administrative 
turnover. The CDE could require the DSET with the school site council to submit an 
explanation as to why the schools did not fully utilize the funds, how the accumulation in 
funds will be effectively utilized in the future, and what - if any - implications this has 
for the Action Plan. 

4. Redesign Annual Report: Collect data necessary to monitor assurances 
and school progress, and review on a regular basis. 

As described above, we recommend enhanced monitoring, and an important step in this 
direction is the modification of the current data collection under this program. While our 
survey collected respondent perceptions about key program components such as the 
external provider and district support, the fact that the evaluation was conducted at the 
end of (and even after) the program made collecting reliable measurements of 
implementation fidelity a challenge. We encourage the state to learn from evaluations of 
CSR model providers, such as High Schools That Work, that use ongoing survey 
measures to assess the extent to which participating schools are implementing the model 
with fidelity and how that relates to student outcomes. We recommend that the CDE 
redesign the Annual Report as a carefully constructed survey instrument that will provide 
indicators of implementation which can then be used, with other measures, to monitor 
schools as well as assess the relationship between implementation and student outcomes. 

The research team did not find the current Annual Report data to be particularly helpful 
in evaluating the program, nor had these data been analyzed in any systematic way prior 
to this evaluation. Eor instance, counts of parents or teachers alone are not helpful in 
measuring the degree of parental involvement or teacher training in relation to other 
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schools. These would need to be converted to percentages to determine the degree of 
parental involvement or trained teachers. 

In conducting the principal analysis presented in Chapter 4, we identified variations in the 
way the years of experience were recorded, with several observations of zero. As we 
cannot verify this information against other statewide data sources (e.g., CBEDS), we are 
unable to determine if it meant the school was currently without a principal, or whether 
principals were at the site for less than a full year. To enhance the reliability of this data, 
questions of this type should be modified so that the current year clearly counts as one (as 
is done in the national Schools and Staffing Survey, SASS), or provide space to enter 
months as well. 

Furthermore, there is evidence that the information is not being updated on a yearly basis, 
or is inaccurate. Taking the principal question as an example, there are records that show 
declines in years at the current site (i.e., 4 years at the site in one year, and 3 years the 
next), or records that show the same information across all three years. 

Perhaps the greatest concern, however, is the high percentage of missing data, which may 
be an indicator of a lack of district or school focus on the HPSGP, or other organizational 
issues. For the years of principal experience at the school site, 15 percent of the HP Only 
schools were missing data for this question in either 2002-03 or 2003-04. As of June 
2007, well over a third of the HP Only schools were missing 2005-06 Annual Report data 
altogether. 

Given the degree of missing data and questions regarding the validity of some 
information, the CDE should review the data annually to ensure completeness and 
accuracy. Changes to the Annual Report data collection could make the data more 
powerful and meaningful for monitoring HPSGP schools and districts. 

5 . Ensure predictable funding: The timing of the funds should be carefully 
considered for the next cohort, with explicit timelines to allow for effective 
school planning and clear expectations regarding a transition phase prior 
to program completion. 

The state and districts should provide clear directives and assurances as to when the funds 
will arrive at the school, how much, and with what degree of flexibility in carry-over. 
Districts with sufficient resources should support schools in implementing the program 
(e.g., allow schools to plan in the spring/summer) when state funds are delayed, and 
schools should be allowed time extensions in meeting their performance targets if the 
funds do not arrive at the school on time. For instance, if resources do not arrive at the 
school until mid-year, it may be unreasonable to expect that substantial academic growth 
will be realized through the program in that year. Or, perhaps districts could be assured in 
some binding way regarding the state’s commitment to forward these funds and be 
directed to fund the school from other monies in the meantime. 

As the achievement analysis shows, sustainability of realized gains is questionable for the 
HPSGP, and this was the case also with the IFUSP evaluation. To facilitate the 
continuation of reform, the CDE should provide clear expectations about a transition 
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phase. For instance, districts and schools (through the external provider and District- 
School Liaison Team) should submit a transition plan at the beginning of the third year of 
implementation. This plan would assess the reforms/changes attributed to HPSGP funds, 
identify which strategies have been most effective, and identify resources necessary (e.g., 
financial and personnel) to allow the schools to continue key strategies beyond the 
HPSGP. This recommendation, however, is based on the premise that there is a clear 
“end year” - which was problematic with the first cohort (in that schools were not 
notified of the fourth year funding until the fall). One district respondent called for three 
years of full funding, followed by two years of partial funding to facilitate alignment of 
budgets and discussions between schools and districts about sustainability issues. 

While we see the merits of a set funding amount over the course of the grant (e.g., same 
total amount across three or four years regardless of enrollment changes) which may 
encourage more effective planning, the state may want to consider modifications in the 
funding amount if schools exhibit a dramatic increase (or decrease) in school enrollment. 

For the second cohort of HPSGP schools, CDE has prohibited annual carry-over. We 
strongly recommend that the CDE reconsider this restriction. While we observed 
considerable carry-over in all years of the program, our case studies suggested that carry- 
over was an indicator of more systemic problems, such as disruption in school leadership. 
As an alternative, we encourage closer monitoring of carry-over, such as requiring 
schools with substantial carry-over to submit an explanation of the reasons and the 
implications for future planning. 

6. Ensure a supply of qualified external providers statewide, consistently 
describe the nature and duration of their role, and add measurements of 
their effectiveness to the program. 

Study respondents expressed concerns regarding the overall supply of qualified external 
providers. If this component is required as part of the HPSGP, the state has an obligation 
to be more proactive in ensuring an adequate and qualified supply. If the state does not 
have the capacity to develop this pool, then perhaps this component should not be 
required, or alternative options should be allowed. 

In addition, a number of school respondents reported the external provider component as 
vaguely defined. This component also showed substantial variation in implementation. 
Although the external provider role is only required in legislation for the development of 
the Action Plan, it is further described in the second cohort guidelines as to “provide 
ongoing technical assistance to the school site administrative and teaching staff.” This 
language seems to imply a relationship with the external provider for the duration of the 
grant. The requirement should be fully clarified and the supporting language made as 
consistent as possible. Establishing annual activities for external providers, such as 
required meetings with the DSET and joint reports to be submitted to the CDE (e.g., for 
schools that do not make their growth targets) may encourage greater consistency and 
heightened presence of the external provider in the school reform process. 

East, the regular cycle of the continuous improvement process described in the second 
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cohort guidelines should include an assessment of the effectiveness of the external 
provider, as currently there appears to be no accountability for these individuals who 
share a large responsibility in assisting the lowest-performing schools in the state. 

One indicator of effectiveness would simply be whether the school makes progress. This 
should clearly not be the sole criterion, however. Beyond this, ratings on the part of those 
being assisted in regard to what the provider actually did, whether this was perceived as 
helpful, and whether they would recommend them to other similar schools might be 
considered. Exactly who has authority over external providers seems unclear, and it may 
not be possible for the CDE to provide this form of oversight. If true, perhaps these types 
of assessments could be conducted independently under contract to the state, or 
legislation may be altered to clarify what measures are in place to assure that external 
providers are actually assisting schools. 

It is also not clear if the participating schools are required to replace external providers if 
the provider ceases support (e.g., retires) or is ineffective, or if the relationship is 
mutually terminated. Given the variability observed with this component and the 
importance that this role plays in the HPSGP (as described by the legislation), we believe 
that it is critical to provide explicit directives regarding this role, including an assessment 
of effectiveness. 



Conclusion 

On average, the state’s lowest-performing schools progressed during the period of 
HPSGP implementation. Although the schools participating in this program did not show 
gains that statistically differ from non-participating schools, all of the schools — as well 
as the state — deserve credit for their advances. The findings from this evaluation should 
not in any way detract from these accomplishments. 

The challenge facing the state’s lowest-performing schools are daunting. Many of the 
educators who participated in the site visits and surveys convinced us of their dedication 
and determination in producing a brighter future for their students. It may simply be that 
the HPSGP was not enough. Ongoing systems of supplemental fiscal resources, selective 
staff placement, and other support are needed to substantially impact student outcomes in 
the state’s most challenged schools. 

Given the primary purpose of the program, some may say that the finding of no 
substantial difference in student performance between HPSGP and comparison schools is 
the only result that matters. As this is the third study issued on behalf of the state showing 
virtually no return in terms of enhanced student performance from the HPSGP and its 
predecessor lEUSP, the question of whether to continue to invest in HPSGP-type 
interventions should be carefully considered by policy makers. Issues related to the need 
to improve student performance in the state’s most challenged schools will not go away 
regardless of the future of the HPSGP. 

We recommend that the state’s commitment to low-performing schools not be 
diminished, but enhanced and re-directed. Because the current investments have not fully 
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yielded the desired results, the need for a bolstered state commitment to equal educational 
opportunities for all children in California is perhaps greater than ever. 
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